Sep 09 20:09:04.884915 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:09:04.884920 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:09:04.884927 ip-10-0-129-45 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:09:04.885280 ip-10-0-129-45 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:09:14.928522 ip-10-0-129-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:09:14.928537 ip-10-0-129-45 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 033325eef356433ebf68babde8ad5637 --
Sep 09 20:11:07.055386 ip-10-0-129-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:07.418895 ip-10-0-129-45 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:07.418895 ip-10-0-129-45 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:07.418895 ip-10-0-129-45 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:07.418895 ip-10-0-129-45 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:07.418895 ip-10-0-129-45 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:07.419804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.419714    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:07.423553 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423534    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.423553 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423553    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423557    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423561    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423564    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423567    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423569    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423572    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423575    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423577    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423582    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423585    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423588    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423597    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423600    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423602    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423605    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423607    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423610    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423612    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423614    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.423621 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423617    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423620    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423623    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423626    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423628    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423632    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423635    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423637    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423640    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423643    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423645    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423648    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423650    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423653    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423655    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423658    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423660    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423662    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423665    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.424116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423668    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423670    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423673    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423675    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423677    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423680    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423684    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423686    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423688    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423690    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423693    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423695    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423697    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423700    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423702    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423707    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423710    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423712    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423716    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423718    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.424586 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423722    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423724    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423728    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423732    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423735    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423738    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423740    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423743    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423745    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423748    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423750    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423753    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423755    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423758    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423760    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423763    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423765    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423768    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423771    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423774    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.425074 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423776    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423779    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423782    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423786    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423789    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423793    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423798    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423803    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423806    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423809    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423812    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423815    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423817    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423820    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423823    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423825    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423827    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423830    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.423833    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.425676 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424664    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424673    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424676    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424679    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424682    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424685    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424689    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424693    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424695    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424698    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424701    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424706    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424710    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424714    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424717    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424719    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424722    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424725    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424728    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424731    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.426420 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424733    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424736    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424739    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424742    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424744    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424747    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424749    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424752    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424755    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424757    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424760    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424762    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424765    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424767    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424770    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424772    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424775    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424778    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424781    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.426970 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424784    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424787    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424789    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424792    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424794    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424796    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424799    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424802    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424804    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424807    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424810    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424813    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424815    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424818    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424820    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424824    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424828    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424831    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424833    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.427468 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424836    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424838    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424841    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424843    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424846    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424848    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424851    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424853    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424856    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424858    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424861    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424864    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424870    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424872    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424875    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424877    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424880    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424883    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424885    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424888    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.427951 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424890    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424893    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424896    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424899    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424901    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424904    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424906    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424909    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424911    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424915    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424918    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424921    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424925    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424927    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424930    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424932    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424935    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424937    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424940    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.428476 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424942    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.424945    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.426974    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.426986    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.426990    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.426993    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.426997    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427000    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427003    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427005    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427008    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427012    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427015    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427017    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427020    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427022    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427025    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427027    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427030    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427032    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.428959 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427035    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427038    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427040    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427043    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427045    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427048    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427050    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427053    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427056    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427058    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427061    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427063    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427066    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427069    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427075    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427078    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427081    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427100    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427105    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.429481 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427109    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427113    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427116    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427119    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427123    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427127    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427130    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427133    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427136    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427138    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427141    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427143    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427146    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427148    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427151    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427153    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427156    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427158    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427161    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.429973 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427164    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427167    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427169    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427172    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427175    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427178    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427181    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427184    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427186    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427189    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427192    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427195    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427197    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427200    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427203    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427205    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427208    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427210    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427213    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427215    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.430455 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427218    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427221    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427223    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427225    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427228    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427230    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427234    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427237    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427240    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427242    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427245    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427247    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427250    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427252    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427255    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427257    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427260    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427262    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427265    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427268    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.430940 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427270    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427273    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.427277    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427357    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427394    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427401    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427406    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427411    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427414    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427419    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427423    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427426    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427430    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427433    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427437    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427440    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427443    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427446    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427449    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427452    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427456    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427460    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427463    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:07.431478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427466    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427469    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427473    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427476    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427479    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427482    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427486    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427489    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427492    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427495    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427499    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427502    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427506    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427509    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427512    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427514    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427517    2579 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427520    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427525    2579 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427527    2579 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427530    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427533    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427536    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427539    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:07.432037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427542    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427545    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427548    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427551    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427554    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427556    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427560    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427562    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427565    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427568    2579 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427572    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427574    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427578    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427581    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427584    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427587    2579 flags.go:64] FLAG: --help="false"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427590    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427593    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427596    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427599    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427602    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427605    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427608    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:07.432631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427611    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427613    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427616    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427620    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427623    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427625    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427628    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427631    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427634    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427637    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427639    2579 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427642    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427645    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427648    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427658    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427661    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427664    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427667    2579 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427669    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427673    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427675    2579 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427678    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427685    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427688    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427693    2579 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:07.433229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427696    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427699    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427702    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427705    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427708    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427710    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427713    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427725    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427728    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427730    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427733    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427736    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427741    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427744    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427747    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427750    2579 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427753    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427756    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02412706a2f702fab"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427759    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427762    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427765    2579 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427768    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427771    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427774    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:07.433848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427777    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427780    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427783    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427786    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427789    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427792    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427795    2579 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427797    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427801    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427803    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427806    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427809    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427812    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427815    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427818    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427821    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427823    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427826    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427829    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427832    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427834    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427837    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427842    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427845    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427848    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:07.434506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427851    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427854    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427859    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427862    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427865    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427868    2579 flags.go:64] FLAG: --v="2"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427872    2579 flags.go:64] FLAG: --version="false"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427876    2579 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427880    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.427884    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428002    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428006    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428009    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428012    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428016    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428019    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428022    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428025    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428027    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428030    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428033    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.435177 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428035    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428039    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428043    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428045    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428048    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428051    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428055    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428058    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428061    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428064    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428067    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428070    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428073    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428076    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428079    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428081    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428105    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428108    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428111    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.435687 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428114    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428119    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428121    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428124    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428126    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428129    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428133    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428135    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428138    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428141    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428143    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428146    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428148    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428151    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428153    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428155    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428158    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428161    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428163    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428166    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.436270 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428168    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428170    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428173    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428175    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428178    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428180    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428184    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428187    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428190    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428192    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428195    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428198    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428200    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428203    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428206    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428208    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428211    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428213    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428217    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428219    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428222    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.436779 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428224    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428228    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428231    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428234    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428237    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428239    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428242    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428245    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428247    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428250    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428252    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428254    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428257    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428259    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428262    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428264    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428267    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428270    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428273    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.437347 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428275    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428278    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428280    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428283    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428285    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428288    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428290    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428293    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.428295    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.428301    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.436522    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.436541    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.436555    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.436561    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436624    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.437830 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436630    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436633    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436636    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436639    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436642    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436645    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436647    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436650    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436653    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436655    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436658    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436662    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436666    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436669    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436672    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436675    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436677    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436680    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436683    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.438260 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436686    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436689    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436692    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436695    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436698    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436701    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436703    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436706    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436709    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436712    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436714    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436717    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436721    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436724    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436726    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436729    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436731    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436734    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436736    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436739    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.438727 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436741    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436743    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436746    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436748    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436751    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436753    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436756    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436758    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436761    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436763    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436766    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436768    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436770    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436773    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436775    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436778    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436782    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436785    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436789    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436792    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.439254 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436796    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436799    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436801    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436804    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436807    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436809    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436812    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436815    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436817    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436820    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436822    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436825    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436828    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436830    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436833    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436835    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436838    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436840    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436843    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436845    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.439745 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436848    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436851    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436853    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436855    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436858    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436860    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436863    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436866    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436868    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436871    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436873    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436876    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436879    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436882    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436885    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436887    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436890    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436893    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.440291 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.436895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.436900    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437013    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437018    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437021    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437024    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437026    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437029    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437031    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437034    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437036    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437039    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437041    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437044    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437047    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:07.440863 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437049    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437052    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437054    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437057    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437059    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437062    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437064    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437067    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437070    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437072    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437075    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437078    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437081    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437106    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437111    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437114    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437117    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437120    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437122    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437124    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:07.441590 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437127    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437130    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437133    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437135    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437138    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437140    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437143    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437145    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437148    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437151    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437155    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437158    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437162    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437167    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437170    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437174    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437178    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437182    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437185    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437187    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:07.442160 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437190    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437192    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437195    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437199    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437203    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437206    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437209    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437211    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437213    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437216    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437221    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437223    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437226    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437229    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437233    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437236    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437238    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437241    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:07.442761 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437244    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437247    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437249    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437252    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437255    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437258    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437260    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437263    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437265    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437267    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437270    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437272    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437274    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437277    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437279    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437282    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437285    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437288    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437292    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437294    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:07.443256 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437297    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437299    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437301    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437305    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437309    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437312    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:07.437314    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.437319    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.437453    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.439631    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.440480    2579 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.440606    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.440659    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:07.443777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.442924    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:07.446299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.446281    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:07.461192 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.461170    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:07.466206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.466187    2579 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:07.467131 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.467106    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:07.471467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.471446    2579 fs.go:135] Filesystem UUIDs: map[213c4f3f-5b2b-4b87-94ed-5f70fb8cbd5a:/dev/nvme0n1p4 2c6e0d8b-2cee-429f-9077-f8e6d5049ccf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:11:07.471552 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.471466    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:07.474394 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.474376    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:07.476453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.476340    2579 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:07.475045946 +0000 UTC m=+0.311360400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099984 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21bf3da4ded04d8a427fa0988b765c SystemUUID:ec21bf3d-a4de-d04d-8a42-7fa0988b765c BootID:033325ee-f356-433e-bf68-babde8ad5637 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:4f:78:b6:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:4f:78:b6:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:85:3f:99:f3:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:07.476453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.476452    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:07.476576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.476548    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:07.480045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.480020    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:07.480201 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.480048    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:07.481341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.481330    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:07.481380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.481343    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:07.481380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.481351    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:07.482063 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.482053    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:07.483641 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.483630    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:07.487783 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487771    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:07.487911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487902    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:07.487949 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487934    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:07.487979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487959    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:07.487979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487973    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:07.488061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487987    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:07.488061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.487998    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:07.488442 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.488429    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:07.488525 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.488450    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:07.490487 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.490457    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:07.493241 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.493225    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:07.493301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.493273    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:07.494132 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494121    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:07.494173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494139    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:07.494173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494145    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:07.494173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494151    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:07.494173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494159    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:07.494173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494166    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494178    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494183    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494191    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494197    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494205    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494210    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494235    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:07.494304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494240    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:07.494975 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.494955    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kqng"
Sep 09 20:11:07.498173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.498156    2579 server.go:1268] "Started kubelet"
Sep 09 20:11:07.498257 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.498195    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:07.498257 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.498229    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:07.498346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.498280    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:07.498696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.498676    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:07.498841 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.498816    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:07.498935 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.498912    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:07.499150 ip-10-0-129-45 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:07.502229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.502203    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:07.502328 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.502259    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kqng"
Sep 09 20:11:07.503189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.503172    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:07.504239 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.504217    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:07.505852 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.505000    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18d3c0139ed5c713  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:ip-10-0-129-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-09-09 20:11:07.497940755 +0000 UTC m=+0.334255208,LastTimestamp:2026-09-09 20:11:07.497940755 +0000 UTC m=+0.334255208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}"
Sep 09 20:11:07.507268 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.507250    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:07.507351 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.507272    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:07.507397 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.507347    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.507998 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.507981    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:07.508137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.507981    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:07.508211 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508141    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:07.508211 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508165    2579 factory.go:55] Registering systemd factory
Sep 09 20:11:07.508211 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508181    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:07.508339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508275    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:07.508339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508283    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:07.508790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508631    2579 factory.go:153] Registering CRI-O factory
Sep 09 20:11:07.508790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508646    2579 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:07.508790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508693    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:07.508790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508709    2579 factory.go:103] Registering Raw factory
Sep 09 20:11:07.508790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.508725    2579 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:07.509145 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.509129    2579 manager.go:349] Starting recovery of all containers
Sep 09 20:11:07.522683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.522646    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:07.525347 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.525316    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.525851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.525836    2579 manager.go:354] Recovery completed
Sep 09 20:11:07.527186 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.527080    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:11:07.530048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.530035    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.532220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532192    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.532283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532226    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.532283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532236    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.532845 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532833    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.532902 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532837    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:07.532950 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532903    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:07.532950 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.532941    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:07.533562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.533545    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.533645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.533576    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.533645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.533590    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.535015 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.535002    2579 policy_none.go:49] "None policy: Start"
Sep 09 20:11:07.535064 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.535018    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:07.535064 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.535028    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:07.537072 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.537054    2579 policy_none.go:47] "Start"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.578673    2579 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.578717    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.578727    2579 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.579008    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.579169    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.579266    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.579331    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.579337    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.579936    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:07.591948 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.579992    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.603652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.603626    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:07.605055 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.605035    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:07.605189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.605101    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:07.605189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.605125    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:07.605189 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.605170    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:07.607141 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.607124    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:07.680355 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.680260    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.681299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.681280    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.681422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.681312    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.681422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.681323    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.681422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.681348    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.689627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.689597    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.689736 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.689630    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-45.ec2.internal\": node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.703030 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.703004    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.706077 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.706061    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Sep 09 20:11:07.706183 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.706173    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.707041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.707025    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.707134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.707056    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.707134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.707066    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.709292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.709276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.709345 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.709300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.709490 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.709476    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.709579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.709567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.709624 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.709595    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.710262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710237    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.710262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710251    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.710379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710268    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.710379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710279    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.710379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710297    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.710379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.710309    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.712523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.712508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.712576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.712538    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:07.713273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.713255    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:07.713346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.713284    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:07.713346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.713298    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:07.737946 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.737923    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.742401 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.742382    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.803874 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.803837    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.809608 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.809582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.809728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.809615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.809728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.809643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.809728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.809692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.809851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.809738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/729b4e6e5ecfd9ee4a0660c00d9444b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"729b4e6e5ecfd9ee4a0660c00d9444b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.904161 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:07.904109    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:07.910424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.910397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:07.910520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:07.910467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034d32653d0ee36541005678f807d7f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"1034d32653d0ee36541005678f807d7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:08.004961 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.004875    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:08.041393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.041367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:08.045347 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.045315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:08.105408 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.105372    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:08.205976 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.205947    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:08.306588 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.306508    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:08.407012 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.406987    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Sep 09 20:11:08.418073 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.418042    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:08.440804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.440761    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:08.441302 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.440962    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:08.441302 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.440991    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:08.441302 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.440991    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:08.489129 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.489080    2579 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:08.506903 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.506876    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:08.507456 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.507439    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:08.507529 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.507461    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:08.507916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.507889    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wqftt","kube-system/konnectivity-agent-q8jz5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442","openshift-dns/node-resolver-xt4vg","openshift-multus/multus-sffbl","openshift-network-diagnostics/network-check-target-2tw8s","openshift-ovn-kubernetes/ovnkube-node-z9m2s","openshift-cluster-node-tuning-operator/tuned-9bjqp","openshift-image-registry/node-ca-65pnk","openshift-multus/multus-additional-cni-plugins-b2qwb","openshift-multus/network-metrics-daemon-dsvjt","openshift-network-operator/iptables-alerter-t992m"]
Sep 09 20:11:08.509886 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.509853    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:08 +0000 UTC" deadline="2028-05-26 01:23:57.71447236 +0000 UTC"
Sep 09 20:11:08.509886 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.509883    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14981h12m49.204591907s"
Sep 09 20:11:08.510518 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.510507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.510583 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.510567    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:08.512617 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.512600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.513511 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.513495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-kubelet-config\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.513593 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.513521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-dbus\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.513593 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.513547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.514928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.514899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.516060 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.516037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:08.516383 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.516366    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:08.516927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.516748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-88w5b\""
Sep 09 20:11:08.517376 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.517273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:08.517376 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.517320    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.517533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.517407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-52qrk\""
Sep 09 20:11:08.517533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.517430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.518002 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.517985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.519737 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.519720    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.519838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.519789    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:08.519838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.519811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p5b76\""
Sep 09 20:11:08.519933 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.519849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.519974 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.519949    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Sep 09 20:11:08.520729 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.520709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.522218 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.522200    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:08.522877 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.522858    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:08.522954 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.522866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.523052 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.523039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:08.523124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.523039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v784q\""
Sep 09 20:11:08.523124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.523117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:08.523201 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.523187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:08.523265 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.523253    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.526170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.526158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.528352 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.528330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.528783 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.528763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:08.528864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.528812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.529916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.529690    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:08.529916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.529698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jcwc5\""
Sep 09 20:11:08.529916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.529718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:08.529916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.529759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:08.529916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.529797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.530237 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.530030    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:08.530703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.530689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.531009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.530899    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.531009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.530913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.531197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.531024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djfkb\""
Sep 09 20:11:08.532758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.532741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.532931 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.532913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.533005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.532984    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:08.533247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.533230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.533325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.533267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qrnv2\""
Sep 09 20:11:08.535245 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.535228    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:08.535346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.535244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:08.535555 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.535538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-48d5m\""
Sep 09 20:11:08.535628 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.535588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.535697 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.535673    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:08.535927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.535912    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8pgbn"
Sep 09 20:11:08.538216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.538198    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"]
Sep 09 20:11:08.538295 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.538230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Sep 09 20:11:08.538295 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.538291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.540627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.540611    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.540891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.540875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.540947 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.540883    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:08.541375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.541361    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p2djg\""
Sep 09 20:11:08.543935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.543919    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8pgbn"
Sep 09 20:11:08.609523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.609498    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:08.613721 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.613851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.613851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-systemd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.613851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-kubelet-config\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.613851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-dbus\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.613851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-slash\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-kubelet\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.613873    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-conf\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-system-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-kubelet-config\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-k8s-cni-cncf-io\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.613949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.113925317 +0000 UTC m=+1.950239774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.613993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-conf-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7b059ae5-251d-4a2b-a6e2-81571644fa39-dbus\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:08.614068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-netns\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43b73432-d77c-4330-b46f-4d25a13a610d-host-slash\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-multus\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-systemd\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fh4d\" (UniqueName: \"kubernetes.io/projected/872fbedf-f91b-469e-8cc7-f66d8d736696-kube-api-access-4fh4d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-agent-certs\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-tmp-dir\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-daemon-config\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-var-lib-kubelet\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-socket-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-device-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-sys-fs\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.614610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysconfig\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrfwv\" (UniqueName: \"kubernetes.io/projected/c99e68cf-f223-4525-83d4-2753c3a7035f-kube-api-access-xrfwv\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-bin\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-env-overrides\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-hosts-file\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-node-log\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnmv\" (UniqueName: \"kubernetes.io/projected/43b73432-d77c-4330-b46f-4d25a13a610d-kube-api-access-sgnmv\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-konnectivity-ca\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-tmp\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-system-cni-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-registration-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.614988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-script-lib\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.615339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d10fd35-665c-41d1-9d16-56bb815fc8b6-host\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2xc\" (UniqueName: \"kubernetes.io/projected/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-kube-api-access-rb2xc\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-bin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-multus-certs\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvs2v\" (UniqueName: \"kubernetes.io/projected/a13d5d69-cb99-4972-a456-f3d71e00d5fe-kube-api-access-wvs2v\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbtm\" (UniqueName: \"kubernetes.io/projected/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kube-api-access-5sbtm\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-var-lib-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-log-socket\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-kubernetes\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-host\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-systemd-units\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-etc-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-ovn\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-netd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovn-node-metrics-cert\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvpv\" (UniqueName: \"kubernetes.io/projected/3c9de731-69fd-49ab-8111-3b88aaef3bcf-kube-api-access-6hvpv\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-etc-kubernetes\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-kubelet\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-config\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43b73432-d77c-4330-b46f-4d25a13a610d-iptables-alerter-script\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-lib-modules\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpvn\" (UniqueName: \"kubernetes.io/projected/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-kube-api-access-zwpvn\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d10fd35-665c-41d1-9d16-56bb815fc8b6-serviceca\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-hostroot\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-tuned\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-cnibin\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.615994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk9t\" (UniqueName: \"kubernetes.io/projected/8d10fd35-665c-41d1-9d16-56bb815fc8b6-kube-api-access-vsk9t\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-socket-dir-parent\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-modprobe-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-sys\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.616984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cnibin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cni-binary-copy\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-netns\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.616984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.616151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-run\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.705003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.703326    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:08.717078 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovn-node-metrics-cert\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717078 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvpv\" (UniqueName: \"kubernetes.io/projected/3c9de731-69fd-49ab-8111-3b88aaef3bcf-kube-api-access-6hvpv\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-etc-kubernetes\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-kubelet\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-config\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43b73432-d77c-4330-b46f-4d25a13a610d-iptables-alerter-script\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-lib-modules\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-etc-kubernetes\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-kubelet\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpvn\" (UniqueName: \"kubernetes.io/projected/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-kube-api-access-zwpvn\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d10fd35-665c-41d1-9d16-56bb815fc8b6-serviceca\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-hostroot\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-tuned\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-cnibin\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-lib-modules\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-hostroot\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-cnibin\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717459    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk9t\" (UniqueName: \"kubernetes.io/projected/8d10fd35-665c-41d1-9d16-56bb815fc8b6-kube-api-access-vsk9t\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-socket-dir-parent\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-modprobe-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-sys\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cnibin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cni-binary-copy\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-netns\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-run\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.717890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-config\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-sys\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-socket-dir-parent\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.717941    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-systemd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-netns\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-slash\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.717998    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.217982655 +0000 UTC m=+2.054297115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cnibin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43b73432-d77c-4330-b46f-4d25a13a610d-iptables-alerter-script\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-kubelet\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-conf\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-system-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.717834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d10fd35-665c-41d1-9d16-56bb815fc8b6-serviceca\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-modprobe-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.718709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-slash\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-k8s-cni-cncf-io\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-conf-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-run\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-conf\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-system-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-netns\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-conf-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-run-netns\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-systemd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-kubelet\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43b73432-d77c-4330-b46f-4d25a13a610d-host-slash\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-k8s-cni-cncf-io\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-multus\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-cni-dir\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43b73432-d77c-4330-b46f-4d25a13a610d-host-slash\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.720936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-systemd\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fh4d\" (UniqueName: \"kubernetes.io/projected/872fbedf-f91b-469e-8cc7-f66d8d736696-kube-api-access-4fh4d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-multus\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysctl-d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-systemd\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-agent-certs\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-tmp-dir\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-daemon-config\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-var-lib-kubelet\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-socket-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-var-lib-kubelet\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-device-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-device-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-sys-fs\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:08.721688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysconfig\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrfwv\" (UniqueName: \"kubernetes.io/projected/c99e68cf-f223-4525-83d4-2753c3a7035f-kube-api-access-xrfwv\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-tmp-dir\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-socket-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.718982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-sys-fs\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-bin\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-bin\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-sysconfig\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-env-overrides\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-hosts-file\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-node-log\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnmv\" (UniqueName: \"kubernetes.io/projected/43b73432-d77c-4330-b46f-4d25a13a610d-kube-api-access-sgnmv\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-konnectivity-ca\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-tmp\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.722273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-node-log\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-hosts-file\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-system-cni-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-registration-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-script-lib\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d10fd35-665c-41d1-9d16-56bb815fc8b6-host\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2xc\" (UniqueName: \"kubernetes.io/projected/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-kube-api-access-rb2xc\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-bin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-env-overrides\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-system-cni-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-multus-certs\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvs2v\" (UniqueName: \"kubernetes.io/projected/a13d5d69-cb99-4972-a456-f3d71e00d5fe-kube-api-access-wvs2v\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94f2938f-6bf5-408b-b0ff-e5af52021aa8-registration-dir\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.722793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-run-multus-certs\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d10fd35-665c-41d1-9d16-56bb815fc8b6-host\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.719916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c99e68cf-f223-4525-83d4-2753c3a7035f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbtm\" (UniqueName: \"kubernetes.io/projected/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kube-api-access-5sbtm\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a13d5d69-cb99-4972-a456-f3d71e00d5fe-host-var-lib-cni-bin\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-var-lib-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovnkube-script-lib\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-multus-daemon-config\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c99e68cf-f223-4525-83d4-2753c3a7035f-cni-binary-copy\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-log-socket\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-kubernetes\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-var-lib-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-host\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-log-socket\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-systemd-units\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-kubernetes\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.723341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-etc-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/872fbedf-f91b-469e-8cc7-f66d8d736696-host\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-ovn\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-systemd-units\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-etc-openvswitch\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-netd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-run-ovn\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c9de731-69fd-49ab-8111-3b88aaef3bcf-host-cni-netd\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-konnectivity-ca\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.720742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a13d5d69-cb99-4972-a456-f3d71e00d5fe-cni-binary-copy\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.721559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-etc-tuned\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.721798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c9de731-69fd-49ab-8111-3b88aaef3bcf-ovn-node-metrics-cert\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.721901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d7ad2e7-1636-49d4-be9a-cade8d13b83e-agent-certs\") pod \"konnectivity-agent-q8jz5\" (UID: \"3d7ad2e7-1636-49d4-be9a-cade8d13b83e\") " pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.723898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.722227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/872fbedf-f91b-469e-8cc7-f66d8d736696-tmp\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.725074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.725022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvpv\" (UniqueName: \"kubernetes.io/projected/3c9de731-69fd-49ab-8111-3b88aaef3bcf-kube-api-access-6hvpv\") pod \"ovnkube-node-z9m2s\" (UID: \"3c9de731-69fd-49ab-8111-3b88aaef3bcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.726322 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.726294    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:08.726423 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.726337    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:08.726423 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.726358    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.726535 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:08.726467    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.226437209 +0000 UTC m=+2.062751672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.727182 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.727135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk9t\" (UniqueName: \"kubernetes.io/projected/8d10fd35-665c-41d1-9d16-56bb815fc8b6-kube-api-access-vsk9t\") pod \"node-ca-65pnk\" (UID: \"8d10fd35-665c-41d1-9d16-56bb815fc8b6\") " pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.727679 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.727638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2xc\" (UniqueName: \"kubernetes.io/projected/67aea7a1-9c4a-4bb3-a71c-70d6cf7da744-kube-api-access-rb2xc\") pod \"node-resolver-xt4vg\" (UID: \"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744\") " pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.727979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.727961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvs2v\" (UniqueName: \"kubernetes.io/projected/a13d5d69-cb99-4972-a456-f3d71e00d5fe-kube-api-access-wvs2v\") pod \"multus-sffbl\" (UID: \"a13d5d69-cb99-4972-a456-f3d71e00d5fe\") " pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.728045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.727983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fh4d\" (UniqueName: \"kubernetes.io/projected/872fbedf-f91b-469e-8cc7-f66d8d736696-kube-api-access-4fh4d\") pod \"tuned-9bjqp\" (UID: \"872fbedf-f91b-469e-8cc7-f66d8d736696\") " pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.728440 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.728413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrfwv\" (UniqueName: \"kubernetes.io/projected/c99e68cf-f223-4525-83d4-2753c3a7035f-kube-api-access-xrfwv\") pod \"multus-additional-cni-plugins-b2qwb\" (UID: \"c99e68cf-f223-4525-83d4-2753c3a7035f\") " pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.728666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.728651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpvn\" (UniqueName: \"kubernetes.io/projected/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-kube-api-access-zwpvn\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:08.729239 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.729216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnmv\" (UniqueName: \"kubernetes.io/projected/43b73432-d77c-4330-b46f-4d25a13a610d-kube-api-access-sgnmv\") pod \"iptables-alerter-t992m\" (UID: \"43b73432-d77c-4330-b46f-4d25a13a610d\") " pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.730065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.730049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbtm\" (UniqueName: \"kubernetes.io/projected/94f2938f-6bf5-408b-b0ff-e5af52021aa8-kube-api-access-5sbtm\") pod \"aws-ebs-csi-driver-node-t4442\" (UID: \"94f2938f-6bf5-408b-b0ff-e5af52021aa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.752526 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.752488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729b4e6e5ecfd9ee4a0660c00d9444b6.slice/crio-5cd495bf49ff0f27d1725a477c9e57655a98162f13cb60f1f4ae7253caf3fd9e WatchSource:0}: Error finding container 5cd495bf49ff0f27d1725a477c9e57655a98162f13cb60f1f4ae7253caf3fd9e: Status 404 returned error can't find the container with id 5cd495bf49ff0f27d1725a477c9e57655a98162f13cb60f1f4ae7253caf3fd9e
Sep 09 20:11:08.752697 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.752682    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1034d32653d0ee36541005678f807d7f.slice/crio-522c7df2902753cf39c62f9e01068f358e4f933fbef74e58bb463b09c6a45c39 WatchSource:0}: Error finding container 522c7df2902753cf39c62f9e01068f358e4f933fbef74e58bb463b09c6a45c39: Status 404 returned error can't find the container with id 522c7df2902753cf39c62f9e01068f358e4f933fbef74e58bb463b09c6a45c39
Sep 09 20:11:08.757354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.757340    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:08.853443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.853350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:08.863191 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.863164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ad2e7_1636_49d4_be9a_cade8d13b83e.slice/crio-49392ce006837e6e3805aca18d05ddd5e67389a99744a724d4e2ead70e0d64d5 WatchSource:0}: Error finding container 49392ce006837e6e3805aca18d05ddd5e67389a99744a724d4e2ead70e0d64d5: Status 404 returned error can't find the container with id 49392ce006837e6e3805aca18d05ddd5e67389a99744a724d4e2ead70e0d64d5
Sep 09 20:11:08.868172 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.868151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442"
Sep 09 20:11:08.876010 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.875988    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f2938f_6bf5_408b_b0ff_e5af52021aa8.slice/crio-41636aa43f5a6fd54194c5bc8eda30b5044452758f826379e4d4ba348d3bd66a WatchSource:0}: Error finding container 41636aa43f5a6fd54194c5bc8eda30b5044452758f826379e4d4ba348d3bd66a: Status 404 returned error can't find the container with id 41636aa43f5a6fd54194c5bc8eda30b5044452758f826379e4d4ba348d3bd66a
Sep 09 20:11:08.892451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.892424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt4vg"
Sep 09 20:11:08.897590 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.897564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sffbl"
Sep 09 20:11:08.901037 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.901015    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aea7a1_9c4a_4bb3_a71c_70d6cf7da744.slice/crio-68f28fd0b08396e95dafd21cc06d44c627d1f1d6620ccf679d4ee35efb14f9ed WatchSource:0}: Error finding container 68f28fd0b08396e95dafd21cc06d44c627d1f1d6620ccf679d4ee35efb14f9ed: Status 404 returned error can't find the container with id 68f28fd0b08396e95dafd21cc06d44c627d1f1d6620ccf679d4ee35efb14f9ed
Sep 09 20:11:08.905797 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.905772    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13d5d69_cb99_4972_a456_f3d71e00d5fe.slice/crio-f8a4163d86693dae44e2e13f2a0f4bb658789c62bf869be1f51ae5c1c651e61f WatchSource:0}: Error finding container f8a4163d86693dae44e2e13f2a0f4bb658789c62bf869be1f51ae5c1c651e61f: Status 404 returned error can't find the container with id f8a4163d86693dae44e2e13f2a0f4bb658789c62bf869be1f51ae5c1c651e61f
Sep 09 20:11:08.916542 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.916523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:08.925637 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.925615    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9de731_69fd_49ab_8111_3b88aaef3bcf.slice/crio-04c62f10e079dd04b75e067ae197c4f69822f6f9653b3bf8cbceb8fc47244d09 WatchSource:0}: Error finding container 04c62f10e079dd04b75e067ae197c4f69822f6f9653b3bf8cbceb8fc47244d09: Status 404 returned error can't find the container with id 04c62f10e079dd04b75e067ae197c4f69822f6f9653b3bf8cbceb8fc47244d09
Sep 09 20:11:08.935664 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.935637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9bjqp"
Sep 09 20:11:08.943530 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.943506    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872fbedf_f91b_469e_8cc7_f66d8d736696.slice/crio-da75d90016e3bfa26ff3daa13bd6f9f72841a599b21122876999d6c4888b257f WatchSource:0}: Error finding container da75d90016e3bfa26ff3daa13bd6f9f72841a599b21122876999d6c4888b257f: Status 404 returned error can't find the container with id da75d90016e3bfa26ff3daa13bd6f9f72841a599b21122876999d6c4888b257f
Sep 09 20:11:08.953848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.953825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65pnk"
Sep 09 20:11:08.963329 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.963306    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d10fd35_665c_41d1_9d16_56bb815fc8b6.slice/crio-bf360f2a0b95c846eac02972a9401c29d92165391a087f320db2f315edb67cf7 WatchSource:0}: Error finding container bf360f2a0b95c846eac02972a9401c29d92165391a087f320db2f315edb67cf7: Status 404 returned error can't find the container with id bf360f2a0b95c846eac02972a9401c29d92165391a087f320db2f315edb67cf7
Sep 09 20:11:08.966287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.966270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b2qwb"
Sep 09 20:11:08.971835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:08.971817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t992m"
Sep 09 20:11:08.975227 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.975203    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99e68cf_f223_4525_83d4_2753c3a7035f.slice/crio-36d08f3f05470a868531e2a5f73c58ab479682c0c51e44aae30d9e4bd7288d2b WatchSource:0}: Error finding container 36d08f3f05470a868531e2a5f73c58ab479682c0c51e44aae30d9e4bd7288d2b: Status 404 returned error can't find the container with id 36d08f3f05470a868531e2a5f73c58ab479682c0c51e44aae30d9e4bd7288d2b
Sep 09 20:11:08.980120 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:11:08.980078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b73432_d77c_4330_b46f_4d25a13a610d.slice/crio-53ff82afb36d46d6c52bac6f347f2685b02c33a230e85e7dcc6840a4dca28997 WatchSource:0}: Error finding container 53ff82afb36d46d6c52bac6f347f2685b02c33a230e85e7dcc6840a4dca28997: Status 404 returned error can't find the container with id 53ff82afb36d46d6c52bac6f347f2685b02c33a230e85e7dcc6840a4dca28997
Sep 09 20:11:09.125073 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.124977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:09.125252 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.125175    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.125252 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.125240    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:10.125225227 +0000 UTC m=+2.961539670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.226130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.226087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:09.226299 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.226238    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.226339 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.226302    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:10.2262795 +0000 UTC m=+3.062593942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.327403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.327361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:09.327622 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.327598    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:09.327697 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.327627    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:09.327697 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.327643    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:09.327818 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.327704    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:10.327685097 +0000 UTC m=+3.163999556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:09.535900 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.535859    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:09.547403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.547333    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:09 +0000 UTC" deadline="2028-04-04 05:24:42.150349543 +0000 UTC"
Sep 09 20:11:09.547403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.547402    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13737h13m32.602952539s"
Sep 09 20:11:09.619568 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.619478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:09.619762 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:09.619734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:09.639951 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.639890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerStarted","Data":"36d08f3f05470a868531e2a5f73c58ab479682c0c51e44aae30d9e4bd7288d2b"}
Sep 09 20:11:09.656554 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.656507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65pnk" event={"ID":"8d10fd35-665c-41d1-9d16-56bb815fc8b6","Type":"ContainerStarted","Data":"bf360f2a0b95c846eac02972a9401c29d92165391a087f320db2f315edb67cf7"}
Sep 09 20:11:09.677234 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.673679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bjqp" event={"ID":"872fbedf-f91b-469e-8cc7-f66d8d736696","Type":"ContainerStarted","Data":"da75d90016e3bfa26ff3daa13bd6f9f72841a599b21122876999d6c4888b257f"}
Sep 09 20:11:09.680276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.680238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"04c62f10e079dd04b75e067ae197c4f69822f6f9653b3bf8cbceb8fc47244d09"}
Sep 09 20:11:09.690537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.690443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt4vg" event={"ID":"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744","Type":"ContainerStarted","Data":"68f28fd0b08396e95dafd21cc06d44c627d1f1d6620ccf679d4ee35efb14f9ed"}
Sep 09 20:11:09.698708 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.698626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"1034d32653d0ee36541005678f807d7f","Type":"ContainerStarted","Data":"522c7df2902753cf39c62f9e01068f358e4f933fbef74e58bb463b09c6a45c39"}
Sep 09 20:11:09.713570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.713523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t992m" event={"ID":"43b73432-d77c-4330-b46f-4d25a13a610d","Type":"ContainerStarted","Data":"53ff82afb36d46d6c52bac6f347f2685b02c33a230e85e7dcc6840a4dca28997"}
Sep 09 20:11:09.728403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.728310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sffbl" event={"ID":"a13d5d69-cb99-4972-a456-f3d71e00d5fe","Type":"ContainerStarted","Data":"f8a4163d86693dae44e2e13f2a0f4bb658789c62bf869be1f51ae5c1c651e61f"}
Sep 09 20:11:09.732619 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.732525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442" event={"ID":"94f2938f-6bf5-408b-b0ff-e5af52021aa8","Type":"ContainerStarted","Data":"41636aa43f5a6fd54194c5bc8eda30b5044452758f826379e4d4ba348d3bd66a"}
Sep 09 20:11:09.740961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.740889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8jz5" event={"ID":"3d7ad2e7-1636-49d4-be9a-cade8d13b83e","Type":"ContainerStarted","Data":"49392ce006837e6e3805aca18d05ddd5e67389a99744a724d4e2ead70e0d64d5"}
Sep 09 20:11:09.743677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:09.743481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerStarted","Data":"5cd495bf49ff0f27d1725a477c9e57655a98162f13cb60f1f4ae7253caf3fd9e"}
Sep 09 20:11:10.039944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.039904    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:10.133853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.133780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:10.134054 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.134016    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:10.134119 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.134087    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:12.134068587 +0000 UTC m=+4.970383033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:10.234383 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.234342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:10.234573 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.234547    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:10.234658 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.234619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:12.234598585 +0000 UTC m=+5.070913032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:10.335089 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.334996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:10.335276 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.335254    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:10.335276 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.335276    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:10.335404 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.335291    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:10.335404 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.335353    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:12.335334271 +0000 UTC m=+5.171648719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:10.402119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.402082    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:10.548382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.548282    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:09 +0000 UTC" deadline="2028-02-14 22:56:33.862357683 +0000 UTC"
Sep 09 20:11:10.548382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.548345    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12554h45m23.314017171s"
Sep 09 20:11:10.606400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.605578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:10.606400 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.605716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:10.606400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:10.606172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:10.606400 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:10.606283    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:11.608910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:11.608276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:11.608910 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:11.608433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:12.153106 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:12.153054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:12.153294 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.153278    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:12.153388 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.153343    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.153323674 +0000 UTC m=+8.989638124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:12.253684 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:12.253468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:12.253684 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.253642    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:12.253963 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.253709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.253688205 +0000 UTC m=+9.090002673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:12.354406 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:12.354364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:12.354617 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.354533    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:12.354617 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.354554    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:12.354617 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.354567    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:12.354793 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.354630    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:16.354609876 +0000 UTC m=+9.190924319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:12.605661 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:12.605559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:12.605851 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.605724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:12.606192 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:12.606131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:12.606282 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:12.606260    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:13.622457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:13.622422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:13.623880 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:13.623349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:14.605840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:14.605797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:14.605840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:14.605828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:14.606072 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:14.605962    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:14.606118 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:14.606063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:15.605883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:15.605838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:15.606418 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:15.606006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:16.187208 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:16.187165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:16.187405 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.187362    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.187484 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.187439    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:24.187418415 +0000 UTC m=+17.023732863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:16.288528 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:16.287992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:16.288528 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.288169    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.288528 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.288225    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:24.288212113 +0000 UTC m=+17.124526555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:16.389348 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:16.389309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:16.389495 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.389478    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:16.389546 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.389501    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:16.389546 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.389511    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.389610 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.389569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:24.389552801 +0000 UTC m=+17.225867241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.606434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:16.606347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:16.606827 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:16.606367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:16.606827 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.606468    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:16.606827 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:16.606504    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:17.606966 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:17.606932    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:17.611235 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:17.608785    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:18.605982 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:18.605945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:18.605982 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:18.605979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:18.606231 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:18.606080    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:18.606520 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:18.606489    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:19.606049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:19.606014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:19.606453 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:19.606148    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:20.605499 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:20.605462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:20.605693 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:20.605510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:20.605693 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:20.605595    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:20.605693 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:20.605686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:21.605957 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:21.605857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:21.606402 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:21.606032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:22.605732 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:22.605692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:22.605880 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:22.605826    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:22.605940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:22.605879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:22.606006 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:22.605978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:23.605582 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:23.605546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:23.605735 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:23.605679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:24.249761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:24.249726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:24.250203 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.249884    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:24.250203 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.249968    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:40.24994448 +0000 UTC m=+33.086258922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:24.350571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:24.350521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:24.350762 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.350719    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:24.350819 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.350797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:40.350774734 +0000 UTC m=+33.187089203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:24.451342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:24.451304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:24.451514 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.451461    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:24.451514 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.451483    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:24.451514 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.451501    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:24.451654 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.451561    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:40.451541361 +0000 UTC m=+33.287855805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:24.606156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:24.606067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:24.606156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:24.606104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:24.606339 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.606216    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:24.606339 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:24.606322    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:25.606440 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:25.606407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:25.606932 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:25.606556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:26.606238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:26.606204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:26.606427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:26.606204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:26.606427 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:26.606329    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:26.606427 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:26.606418    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:27.607306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:27.607272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:27.607763 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:27.607406    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:28.605594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:28.605558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:28.605594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:28.605591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:28.605791 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:28.605658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:28.605791 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:28.605764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:29.605836 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.605549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:29.606355 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:29.605924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:29.818121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.818069    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="c3b9c00a90eed522b2b4cc155c9ef0919c6eaadf28ef436546e797d2c13443c2" exitCode=0
Sep 09 20:11:29.818256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.818129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"c3b9c00a90eed522b2b4cc155c9ef0919c6eaadf28ef436546e797d2c13443c2"}
Sep 09 20:11:29.819867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.819843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65pnk" event={"ID":"8d10fd35-665c-41d1-9d16-56bb815fc8b6","Type":"ContainerStarted","Data":"f47b1358af9e64fcb46841bdb0c503b60787e9d600d1007181bce8c0791a3297"}
Sep 09 20:11:29.821571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.821551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9bjqp" event={"ID":"872fbedf-f91b-469e-8cc7-f66d8d736696","Type":"ContainerStarted","Data":"4fc0914d52ff2942e83733d6c9eb1ecb539af4830e1d1f0edb7a017e53787d14"}
Sep 09 20:11:29.825027 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"b87fb0e4568d34671b8e4f5846c2b22373327e14b439a4aeb4f7387a07dc5fc6"}
Sep 09 20:11:29.825144 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"5825a95a1cece6afaff41552b5c1c059ea6ad189f0eda7a495ca64e9ad5f65c4"}
Sep 09 20:11:29.825204 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"3cf9c7d8ac0c3868b5df3b0c23049b8f3d9cb43bff724c6bd3a69a1c8d7726ff"}
Sep 09 20:11:29.825241 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"6e257db1e1e73615c3d3131f7f18cda51dca8830c6e1c03270c55eebd2942da3"}
Sep 09 20:11:29.825287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"83fb1d252c6b9a200b84735aaa85704b3d57b3b7517b855421c73d1d46540ae1"}
Sep 09 20:11:29.825334 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.825291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"cd29318099d05bec1337f26bcd6a389052c30650a7e33d92d0a991cf712c35ce"}
Sep 09 20:11:29.826607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.826583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt4vg" event={"ID":"67aea7a1-9c4a-4bb3-a71c-70d6cf7da744","Type":"ContainerStarted","Data":"c2a072bd34ec959fb754e7b39446def4f372a6e3bf3331350544179a7025cbd7"}
Sep 09 20:11:29.828056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.828036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"1034d32653d0ee36541005678f807d7f","Type":"ContainerStarted","Data":"2707a693747ce64d0057070ebf07b446f8f439018b428624b5b847f0e9eab8f9"}
Sep 09 20:11:29.829600 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.829581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sffbl" event={"ID":"a13d5d69-cb99-4972-a456-f3d71e00d5fe","Type":"ContainerStarted","Data":"fc8984744cab184db314a88306e1ce61bfa76e2eef73da995ccb95ae57d35c46"}
Sep 09 20:11:29.831197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.831172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442" event={"ID":"94f2938f-6bf5-408b-b0ff-e5af52021aa8","Type":"ContainerStarted","Data":"a82de821ea685be3cb261bcd4dedd5079618c43bbf7bbcb22759f71043263128"}
Sep 09 20:11:29.833066 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.833043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q8jz5" event={"ID":"3d7ad2e7-1636-49d4-be9a-cade8d13b83e","Type":"ContainerStarted","Data":"489536eba2bb6200a563cde11d4b6f7a1f19a4241f9eb6785817f29cbdd23825"}
Sep 09 20:11:29.834855 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.834832    2579 generic.go:356] "Generic (PLEG): container finished" podID="729b4e6e5ecfd9ee4a0660c00d9444b6" containerID="5b2be708220db38d2ccf9434d333c13e7c9b5957782f575f6bf99dd6f0867c9c" exitCode=0
Sep 09 20:11:29.834950 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.834867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerDied","Data":"5b2be708220db38d2ccf9434d333c13e7c9b5957782f575f6bf99dd6f0867c9c"}
Sep 09 20:11:29.852930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.852877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sffbl" podStartSLOduration=1.627559188 podStartE2EDuration="21.852861328s" totalImagesPullingTime="20.22530214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.852621763 +0000 UTC m=+22.688936227" watchObservedRunningTime="2026-09-09 20:11:29.852861328 +0000 UTC m=+22.689175791"
Sep 09 20:11:29.865630 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.865578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-65pnk" podStartSLOduration=1.794833256 podStartE2EDuration="21.86556228s" totalImagesPullingTime="20.070729024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.865533425 +0000 UTC m=+22.701847887" watchObservedRunningTime="2026-09-09 20:11:29.86556228 +0000 UTC m=+22.701876742"
Sep 09 20:11:29.879327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.879280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xt4vg" podStartSLOduration=1.746288949 podStartE2EDuration="21.879260497s" totalImagesPullingTime="20.132971548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.878942385 +0000 UTC m=+22.715256849" watchObservedRunningTime="2026-09-09 20:11:29.879260497 +0000 UTC m=+22.715574964"
Sep 09 20:11:29.894710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.894652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9bjqp" podStartSLOduration=1.708012149 podStartE2EDuration="21.894633322s" totalImagesPullingTime="20.186621173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.894338018 +0000 UTC m=+22.730652483" watchObservedRunningTime="2026-09-09 20:11:29.894633322 +0000 UTC m=+22.730947786"
Sep 09 20:11:29.923219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.923171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" podStartSLOduration=20.923153651 podStartE2EDuration="20.923153651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.922606287 +0000 UTC m=+22.758920749" watchObservedRunningTime="2026-09-09 20:11:29.923153651 +0000 UTC m=+22.759468113"
Sep 09 20:11:29.938551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:29.938496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q8jz5" podStartSLOduration=1.767831128 podStartE2EDuration="21.938478491s" totalImagesPullingTime="20.170647363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.938015022 +0000 UTC m=+22.774329486" watchObservedRunningTime="2026-09-09 20:11:29.938478491 +0000 UTC m=+22.774792954"
Sep 09 20:11:30.605767 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.605421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:30.605767 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.605447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:30.605767 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:30.605556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:30.605767 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:30.605658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:30.839997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.839748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"729b4e6e5ecfd9ee4a0660c00d9444b6","Type":"ContainerStarted","Data":"00c1f7778d066054cfb9e9768e3daf13188e7ba1f0c367edab4f2e65ac324d2f"}
Sep 09 20:11:30.841613 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.841579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t992m" event={"ID":"43b73432-d77c-4330-b46f-4d25a13a610d","Type":"ContainerStarted","Data":"010f5f4d9513c2fbd609ad759df33744cb0868dd416ff7ddc711ff7e55fccf1f"}
Sep 09 20:11:30.854765 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.854716    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" podStartSLOduration=21.854700998 podStartE2EDuration="21.854700998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:30.854526576 +0000 UTC m=+23.690841054" watchObservedRunningTime="2026-09-09 20:11:30.854700998 +0000 UTC m=+23.691015463"
Sep 09 20:11:30.868652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:30.868563    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t992m" podStartSLOduration=2.7223721640000003 podStartE2EDuration="22.868548236s" totalImagesPullingTime="20.146176072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:30.868140798 +0000 UTC m=+23.704455261" watchObservedRunningTime="2026-09-09 20:11:30.868548236 +0000 UTC m=+23.704862698"
Sep 09 20:11:31.161364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.161320    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:31.592616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.592438    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:31.161344767Z","UUID":"8e01f6ce-18bd-45d2-8bf8-b027295db714","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:31.594200 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.594167    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:31.594200 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.594207    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:31.605527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.605500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:31.605658 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:31.605613    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:31.847035 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.846944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442" event={"ID":"94f2938f-6bf5-408b-b0ff-e5af52021aa8","Type":"ContainerStarted","Data":"e150e76e4d67d72d6aeba4140c9891cae997e0df2eec7733b0ea059d7a0f94bd"}
Sep 09 20:11:31.930309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.930040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:31.930719 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:31.930690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:32.605659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.605562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:32.605659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.605588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:32.605888 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:32.605674    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:32.605888 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:32.605807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:32.853255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.853218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"58f66eff86f15c2880a26c8d79fbacc063d7ee759de217eaebe5b168f900bbc7"}
Sep 09 20:11:32.856110 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.856009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442" event={"ID":"94f2938f-6bf5-408b-b0ff-e5af52021aa8","Type":"ContainerStarted","Data":"f303ca6d8ab8da77c43393e6fb505f84e6de4502843fed5fccf763542e481e0e"}
Sep 09 20:11:32.856293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.856268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:32.856994 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.856974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q8jz5"
Sep 09 20:11:32.872601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:32.872549    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4442" podStartSLOduration=2.191664148 podStartE2EDuration="24.872533574s" totalImagesPullingTime="22.680869426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:32.872365242 +0000 UTC m=+25.708679706" watchObservedRunningTime="2026-09-09 20:11:32.872533574 +0000 UTC m=+25.708848036"
Sep 09 20:11:33.605758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:33.605721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:33.605952 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:33.605857    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:34.605884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.605704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:34.606319 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.605704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:34.606319 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:34.605956    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:34.606319 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:34.606024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:34.863600 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.863559    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="9a045a9ee426ced44e2d6ff3f1bcfbf9a13394f9ba7c1fade4260e925cdf05ea" exitCode=0
Sep 09 20:11:34.863792 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.863638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"9a045a9ee426ced44e2d6ff3f1bcfbf9a13394f9ba7c1fade4260e925cdf05ea"}
Sep 09 20:11:34.867791 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.867768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" event={"ID":"3c9de731-69fd-49ab-8111-3b88aaef3bcf","Type":"ContainerStarted","Data":"bcdf14bf8336b367fd8b0447ef2da4b77adf99cf1b8062f5650bcbfc3e5720b1"}
Sep 09 20:11:34.868040 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.868007    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:34.868133 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.868114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:34.868190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.868184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:34.884444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.884419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:34.884586 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.884516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:11:34.911918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:34.911859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s" podStartSLOduration=6.655449961 podStartE2EDuration="26.911843549s" totalImagesPullingTime="20.256393588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:34.910299735 +0000 UTC m=+27.746614197" watchObservedRunningTime="2026-09-09 20:11:34.911843549 +0000 UTC m=+27.748158011"
Sep 09 20:11:35.606321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:35.606289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:35.606838 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:35.606437    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:36.323061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.323028    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wqftt"]
Sep 09 20:11:36.323272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.323193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:36.323336 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:36.323302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:36.325198 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.325170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tw8s"]
Sep 09 20:11:36.325346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.325280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:36.325408 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:36.325373    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:36.325795 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.325777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsvjt"]
Sep 09 20:11:36.325879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.325841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:36.325941 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:36.325920    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:36.874871 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.874832    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="07bc30217bb6acecf61524d5a9a4d4e98b0650574f238e0b751eb7db62efd007" exitCode=0
Sep 09 20:11:36.875660 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:36.874973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"07bc30217bb6acecf61524d5a9a4d4e98b0650574f238e0b751eb7db62efd007"}
Sep 09 20:11:37.606579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:37.606397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:37.606779 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:37.606677    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:38.605999 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:38.605915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:38.606439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:38.605915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:38.606439 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:38.606062    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:38.606439 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:38.606111    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:38.881837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:38.881761    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="507d8b6a1ab845af31fa3065eb44f3fc0a5abc73be851b0b23e6e7dbac1c4456" exitCode=0
Sep 09 20:11:38.881837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:38.881813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"507d8b6a1ab845af31fa3065eb44f3fc0a5abc73be851b0b23e6e7dbac1c4456"}
Sep 09 20:11:39.606362 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:39.606320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:39.606832 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:39.606447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:40.268776 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:40.268695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:40.269009 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.268871    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:40.269009 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.268947    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret podName:7b059ae5-251d-4a2b-a6e2-81571644fa39 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:12.268926902 +0000 UTC m=+65.105241346 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret") pod "global-pull-secret-syncer-wqftt" (UID: "7b059ae5-251d-4a2b-a6e2-81571644fa39") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:40.369967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:40.369918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:40.370172 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.370078    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:40.370172 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.370163    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs podName:ffa78a3f-5d6e-4689-a436-65a2aafc1e01 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:12.370145691 +0000 UTC m=+65.206460136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs") pod "network-metrics-daemon-dsvjt" (UID: "ffa78a3f-5d6e-4689-a436-65a2aafc1e01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:40.470532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:40.470497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:40.470716 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.470666    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:40.470716 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.470691    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:40.470716 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.470704    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9mv8r for pod openshift-network-diagnostics/network-check-target-2tw8s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:40.470863 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.470762    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r podName:8517e6d9-209b-4478-acaa-882e2d592fb8 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:12.470747885 +0000 UTC m=+65.307062326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9mv8r" (UniqueName: "kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r") pod "network-check-target-2tw8s" (UID: "8517e6d9-209b-4478-acaa-882e2d592fb8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:40.606343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:40.606269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:40.606343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:40.606298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:40.606829 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.606396    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:40.606829 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:40.606484    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:41.009265 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:41.009235    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xt4vg_67aea7a1-9c4a-4bb3-a71c-70d6cf7da744/dns-node-resolver/0.log"
Sep 09 20:11:41.593331 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:41.593303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65pnk_8d10fd35-665c-41d1-9d16-56bb815fc8b6/node-ca/0.log"
Sep 09 20:11:41.605838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:41.605803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:41.606008 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:41.605939    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2tw8s" podUID="8517e6d9-209b-4478-acaa-882e2d592fb8"
Sep 09 20:11:42.605459 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:42.605377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:42.605459 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:42.605395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:42.605925 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:42.605552    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dsvjt" podUID="ffa78a3f-5d6e-4689-a436-65a2aafc1e01"
Sep 09 20:11:42.605925 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:11:42.605683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wqftt" podUID="7b059ae5-251d-4a2b-a6e2-81571644fa39"
Sep 09 20:11:42.894845 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:42.894806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerStarted","Data":"e621932f9d988198c60f3edb63cc61ecabdff4ab4ca9f572a0d9b6075b847378"}
Sep 09 20:11:43.015716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.015687    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeReady"
Sep 09 20:11:43.015908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.015824    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:43.606339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.606308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:11:43.609401 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.609380    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:43.610522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.610502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qbp9b\""
Sep 09 20:11:43.610642 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.610542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:43.900082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.900048    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="e621932f9d988198c60f3edb63cc61ecabdff4ab4ca9f572a0d9b6075b847378" exitCode=0
Sep 09 20:11:43.900262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:43.900121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"e621932f9d988198c60f3edb63cc61ecabdff4ab4ca9f572a0d9b6075b847378"}
Sep 09 20:11:44.605940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.605899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:11:44.606149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.605907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:11:44.608407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.608371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:44.608407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.608368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:44.608407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.608399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8s569\""
Sep 09 20:11:44.906892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.906689    2579 generic.go:356] "Generic (PLEG): container finished" podID="c99e68cf-f223-4525-83d4-2753c3a7035f" containerID="e71ff543e170b649128fa94994ceeaa21cb80dbfb2fd528ea4b8a59cf1ebf50b" exitCode=0
Sep 09 20:11:44.907065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:44.906754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerDied","Data":"e71ff543e170b649128fa94994ceeaa21cb80dbfb2fd528ea4b8a59cf1ebf50b"}
Sep 09 20:11:45.912764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:45.912675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" event={"ID":"c99e68cf-f223-4525-83d4-2753c3a7035f","Type":"ContainerStarted","Data":"f8cefd53ff3caaa623a2ba9c94dd93f316208f463bd4d9f1d3fcdce084c3127e"}
Sep 09 20:11:45.934965 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:11:45.934914    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b2qwb" podStartSLOduration=6.997474969 podStartE2EDuration="37.934895336s" totalImagesPullingTime="30.937420367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:45.933207833 +0000 UTC m=+38.769522297" watchObservedRunningTime="2026-09-09 20:11:45.934895336 +0000 UTC m=+38.771209799"
Sep 09 20:12:02.957201 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.957171    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97fdbc685-p5s4b"]
Sep 09 20:12:02.958997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.958980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:02.961779 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.961755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:12:02.961909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.961800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:12:02.961909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.961817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8nvwp\""
Sep 09 20:12:02.961909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.961800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:12:02.967700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.967675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:12:02.973159 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.973130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97fdbc685-p5s4b"]
Sep 09 20:12:02.982584 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.982559    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pn5rp"]
Sep 09 20:12:02.984437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.984421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:02.987003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.986981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:12:02.987127 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.987024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:12:02.987663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.987643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5th2x\""
Sep 09 20:12:02.988187 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.988166    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rp476"]
Sep 09 20:12:02.990073 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.990058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:02.992612 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.992591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:12:02.992683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.992594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:12:02.992683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.992666    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:12:02.992800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.992619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xqpc7\""
Sep 09 20:12:02.992856 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.992847    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:12:02.995455 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:02.995439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pn5rp"]
Sep 09 20:12:03.001800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.001780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rp476"]
Sep 09 20:12:03.087908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.087876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x28rk"]
Sep 09 20:12:03.089778 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.089762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.092413 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.092391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hmzcc\""
Sep 09 20:12:03.092512 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.092424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:12:03.092512 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.092398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:12:03.092599 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.092526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:12:03.099191 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.099170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x28rk"]
Sep 09 20:12:03.129036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.128994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stcg6\" (UniqueName: \"kubernetes.io/projected/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-api-access-stcg6\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.129036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-bound-sa-token\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/09e49098-9703-4ac2-97a4-66d87c1084da-metrics-tls\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.129233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/09e49098-9703-4ac2-97a4-66d87c1084da-config-volume\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.129233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-crio-socket\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.129233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-image-registry-private-configuration\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-trusted-ca\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldwp\" (UniqueName: \"kubernetes.io/projected/09e49098-9703-4ac2-97a4-66d87c1084da-kube-api-access-4ldwp\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/121425d6-de91-426a-bcee-c44d2adc5f24-ca-trust-extracted\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-installation-pull-secrets\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-data-volume\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.129569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpgt\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-kube-api-access-rvpgt\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e49098-9703-4ac2-97a4-66d87c1084da-tmp-dir\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.129569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-registry-tls\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.129569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.129499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-registry-certificates\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.230889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-cert\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.230889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-crio-socket\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.230889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-image-registry-private-configuration\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-crio-socket\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-trusted-ca\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.230976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldwp\" (UniqueName: \"kubernetes.io/projected/09e49098-9703-4ac2-97a4-66d87c1084da-kube-api-access-4ldwp\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/121425d6-de91-426a-bcee-c44d2adc5f24-ca-trust-extracted\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-installation-pull-secrets\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-data-volume\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpgt\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-kube-api-access-rvpgt\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e49098-9703-4ac2-97a4-66d87c1084da-tmp-dir\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltbg\" (UniqueName: \"kubernetes.io/projected/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-kube-api-access-jltbg\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-registry-tls\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-registry-certificates\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stcg6\" (UniqueName: \"kubernetes.io/projected/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-api-access-stcg6\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-bound-sa-token\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/09e49098-9703-4ac2-97a4-66d87c1084da-metrics-tls\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/09e49098-9703-4ac2-97a4-66d87c1084da-config-volume\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-data-volume\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.231672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/121425d6-de91-426a-bcee-c44d2adc5f24-ca-trust-extracted\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.232190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/09e49098-9703-4ac2-97a4-66d87c1084da-config-volume\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.232190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.231957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-trusted-ca\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.232480 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.232315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e49098-9703-4ac2-97a4-66d87c1084da-tmp-dir\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.232480 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.232430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/121425d6-de91-426a-bcee-c44d2adc5f24-registry-certificates\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.235492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.235472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-registry-tls\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.235610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.235590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.235649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.235617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-installation-pull-secrets\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.235687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.235644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/121425d6-de91-426a-bcee-c44d2adc5f24-image-registry-private-configuration\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.235852 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.235837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/09e49098-9703-4ac2-97a4-66d87c1084da-metrics-tls\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.239130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.239108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldwp\" (UniqueName: \"kubernetes.io/projected/09e49098-9703-4ac2-97a4-66d87c1084da-kube-api-access-4ldwp\") pod \"dns-default-pn5rp\" (UID: \"09e49098-9703-4ac2-97a4-66d87c1084da\") " pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.239260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.239241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stcg6\" (UniqueName: \"kubernetes.io/projected/a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf-kube-api-access-stcg6\") pod \"insights-runtime-extractor-rp476\" (UID: \"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf\") " pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.239686 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.239662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpgt\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-kube-api-access-rvpgt\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.239821 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.239797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/121425d6-de91-426a-bcee-c44d2adc5f24-bound-sa-token\") pod \"image-registry-97fdbc685-p5s4b\" (UID: \"121425d6-de91-426a-bcee-c44d2adc5f24\") " pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.269851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.269817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.293544 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.293503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:03.301592 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.301567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rp476"
Sep 09 20:12:03.333430 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.332872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltbg\" (UniqueName: \"kubernetes.io/projected/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-kube-api-access-jltbg\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.333430 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.332943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-cert\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.337527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.337470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-cert\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.359519 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.359360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltbg\" (UniqueName: \"kubernetes.io/projected/d26c70cd-2e0d-4073-aeb0-5775d8d1983e-kube-api-access-jltbg\") pod \"ingress-canary-x28rk\" (UID: \"d26c70cd-2e0d-4073-aeb0-5775d8d1983e\") " pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.398154 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.398122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x28rk"
Sep 09 20:12:03.558640 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.558534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97fdbc685-p5s4b"]
Sep 09 20:12:03.562723 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:03.562690    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121425d6_de91_426a_bcee_c44d2adc5f24.slice/crio-ce2d795a2e21358ea111587cce4952f1a5e3aaebf74d7023558ef4b8f19ee1f5 WatchSource:0}: Error finding container ce2d795a2e21358ea111587cce4952f1a5e3aaebf74d7023558ef4b8f19ee1f5: Status 404 returned error can't find the container with id ce2d795a2e21358ea111587cce4952f1a5e3aaebf74d7023558ef4b8f19ee1f5
Sep 09 20:12:03.615026 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.614997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pn5rp"]
Sep 09 20:12:03.617975 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.617912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rp476"]
Sep 09 20:12:03.618238 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:03.618216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e49098_9703_4ac2_97a4_66d87c1084da.slice/crio-15f34f008de09c582c47ee936b4b1b3b4bc8442744b1d7b6eb9d5b4974a7e06a WatchSource:0}: Error finding container 15f34f008de09c582c47ee936b4b1b3b4bc8442744b1d7b6eb9d5b4974a7e06a: Status 404 returned error can't find the container with id 15f34f008de09c582c47ee936b4b1b3b4bc8442744b1d7b6eb9d5b4974a7e06a
Sep 09 20:12:03.621600 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:03.621575    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f3a2bb_d41d_4c1c_9ef2_3ca93b5b34cf.slice/crio-e3eb57c1b087cd222ee69245d97366df21a688958258e705e7e411aac6dfc940 WatchSource:0}: Error finding container e3eb57c1b087cd222ee69245d97366df21a688958258e705e7e411aac6dfc940: Status 404 returned error can't find the container with id e3eb57c1b087cd222ee69245d97366df21a688958258e705e7e411aac6dfc940
Sep 09 20:12:03.663424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.663396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x28rk"]
Sep 09 20:12:03.666434 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:03.666406    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26c70cd_2e0d_4073_aeb0_5775d8d1983e.slice/crio-a381b2c8cecd95cf1c44390c85a7c8c9da82934a6d331c25b2c36ea1ac53310f WatchSource:0}: Error finding container a381b2c8cecd95cf1c44390c85a7c8c9da82934a6d331c25b2c36ea1ac53310f: Status 404 returned error can't find the container with id a381b2c8cecd95cf1c44390c85a7c8c9da82934a6d331c25b2c36ea1ac53310f
Sep 09 20:12:03.956908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.956860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pn5rp" event={"ID":"09e49098-9703-4ac2-97a4-66d87c1084da","Type":"ContainerStarted","Data":"15f34f008de09c582c47ee936b4b1b3b4bc8442744b1d7b6eb9d5b4974a7e06a"}
Sep 09 20:12:03.958550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.958520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b" event={"ID":"121425d6-de91-426a-bcee-c44d2adc5f24","Type":"ContainerStarted","Data":"33adb70ef645df421a3db54cb0ad69706f3efd95d1ccfdb1c1ac0c70d5ab5b6b"}
Sep 09 20:12:03.958967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.958564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b" event={"ID":"121425d6-de91-426a-bcee-c44d2adc5f24","Type":"ContainerStarted","Data":"ce2d795a2e21358ea111587cce4952f1a5e3aaebf74d7023558ef4b8f19ee1f5"}
Sep 09 20:12:03.958967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.958637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:03.959861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.959839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x28rk" event={"ID":"d26c70cd-2e0d-4073-aeb0-5775d8d1983e","Type":"ContainerStarted","Data":"a381b2c8cecd95cf1c44390c85a7c8c9da82934a6d331c25b2c36ea1ac53310f"}
Sep 09 20:12:03.961550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.961517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp476" event={"ID":"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf","Type":"ContainerStarted","Data":"afe90b0dc2093b17dc95921e4e2c8114241e327ef2c1592caf538c3ee0e14f64"}
Sep 09 20:12:03.961628 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.961566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp476" event={"ID":"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf","Type":"ContainerStarted","Data":"e3eb57c1b087cd222ee69245d97366df21a688958258e705e7e411aac6dfc940"}
Sep 09 20:12:03.979520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:03.979474    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b" podStartSLOduration=1.979448326 podStartE2EDuration="1.979448326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:03.978267491 +0000 UTC m=+56.814581955" watchObservedRunningTime="2026-09-09 20:12:03.979448326 +0000 UTC m=+56.815762789"
Sep 09 20:12:04.968206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:04.968162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp476" event={"ID":"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf","Type":"ContainerStarted","Data":"79368c24525477e9e5d09dcb7bf2a8f8b81edc57ee6f0a42849e7831354e8abb"}
Sep 09 20:12:05.972260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:05.972220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pn5rp" event={"ID":"09e49098-9703-4ac2-97a4-66d87c1084da","Type":"ContainerStarted","Data":"88aa7b7c7b0e81a08970ea02e665b9ac23c67cc7ec021a1f32c98fb33cedbb12"}
Sep 09 20:12:06.889066 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.888996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9m2s"
Sep 09 20:12:06.977859 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.977819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pn5rp" event={"ID":"09e49098-9703-4ac2-97a4-66d87c1084da","Type":"ContainerStarted","Data":"85c62b7f1b55fa315aa6a004ac2f461853f31ebe019022ef9a7c895f29e31235"}
Sep 09 20:12:06.978343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.977910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:06.979546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.979521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x28rk" event={"ID":"d26c70cd-2e0d-4073-aeb0-5775d8d1983e","Type":"ContainerStarted","Data":"39cf0e21594c16d211b81a1253ba156f7ae365de09e7e0f0c677e4ddd02d236d"}
Sep 09 20:12:06.981677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.981655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rp476" event={"ID":"a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf","Type":"ContainerStarted","Data":"d576c74184673b0b4836dc34896723fba2fc79aa31f4482485bdebc598df5664"}
Sep 09 20:12:06.995281 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:06.995237    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pn5rp" podStartSLOduration=3.374869217 podStartE2EDuration="4.995221091s" totalImagesPullingTime="1.620351874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:06.994394421 +0000 UTC m=+59.830708885" watchObservedRunningTime="2026-09-09 20:12:06.995221091 +0000 UTC m=+59.831535553"
Sep 09 20:12:07.010164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:07.010079    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x28rk" podStartSLOduration=1.148491816 podStartE2EDuration="4.010059867s" totalImagesPullingTime="2.861568051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:07.008816517 +0000 UTC m=+59.845130982" watchObservedRunningTime="2026-09-09 20:12:07.010059867 +0000 UTC m=+59.846374328"
Sep 09 20:12:07.026618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:07.026565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rp476" podStartSLOduration=2.253353718 podStartE2EDuration="5.026547084s" totalImagesPullingTime="2.773193366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:07.025748729 +0000 UTC m=+59.862063217" watchObservedRunningTime="2026-09-09 20:12:07.026547084 +0000 UTC m=+59.862861547"
Sep 09 20:12:12.301188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.301143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:12:12.304227 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.304202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:12.313904 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.313880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7b059ae5-251d-4a2b-a6e2-81571644fa39-original-pull-secret\") pod \"global-pull-secret-syncer-wqftt\" (UID: \"7b059ae5-251d-4a2b-a6e2-81571644fa39\") " pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:12:12.402278 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.402232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:12:12.405297 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.405279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:12.415782 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.415748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa78a3f-5d6e-4689-a436-65a2aafc1e01-metrics-certs\") pod \"network-metrics-daemon-dsvjt\" (UID: \"ffa78a3f-5d6e-4689-a436-65a2aafc1e01\") " pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:12:12.503059 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.503029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:12:12.505923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.505904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:12.515304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.515286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wqftt"
Sep 09 20:12:12.516531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.516515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:12.522888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.522869    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8s569\""
Sep 09 20:12:12.527619 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.527594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mv8r\" (UniqueName: \"kubernetes.io/projected/8517e6d9-209b-4478-acaa-882e2d592fb8-kube-api-access-9mv8r\") pod \"network-check-target-2tw8s\" (UID: \"8517e6d9-209b-4478-acaa-882e2d592fb8\") " pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:12:12.530939 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.530919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dsvjt"
Sep 09 20:12:12.718629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.718596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qbp9b\""
Sep 09 20:12:12.726712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.726692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:12:12.797778 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.797754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wqftt"]
Sep 09 20:12:12.800329 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:12.800300    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b059ae5_251d_4a2b_a6e2_81571644fa39.slice/crio-bf42f42345a4bf6608cd640564a8a0e2a8b8d001f9ac4761101ab52b84644fcd WatchSource:0}: Error finding container bf42f42345a4bf6608cd640564a8a0e2a8b8d001f9ac4761101ab52b84644fcd: Status 404 returned error can't find the container with id bf42f42345a4bf6608cd640564a8a0e2a8b8d001f9ac4761101ab52b84644fcd
Sep 09 20:12:12.810058 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.810032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dsvjt"]
Sep 09 20:12:12.812684 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:12.812660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa78a3f_5d6e_4689_a436_65a2aafc1e01.slice/crio-31898f4a3e2f7a7b10154393d1dc9a68d101725f862a4b58b48047c5f6b0a4cd WatchSource:0}: Error finding container 31898f4a3e2f7a7b10154393d1dc9a68d101725f862a4b58b48047c5f6b0a4cd: Status 404 returned error can't find the container with id 31898f4a3e2f7a7b10154393d1dc9a68d101725f862a4b58b48047c5f6b0a4cd
Sep 09 20:12:12.990872 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:12.990838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2tw8s"]
Sep 09 20:12:12.993520 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:12.993492    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8517e6d9_209b_4478_acaa_882e2d592fb8.slice/crio-d15200c382f03e5149013aea1d4f9dabcb26ffdcc4d8faaa5526d05a4e74aab6 WatchSource:0}: Error finding container d15200c382f03e5149013aea1d4f9dabcb26ffdcc4d8faaa5526d05a4e74aab6: Status 404 returned error can't find the container with id d15200c382f03e5149013aea1d4f9dabcb26ffdcc4d8faaa5526d05a4e74aab6
Sep 09 20:12:13.000316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:13.000292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tw8s" event={"ID":"8517e6d9-209b-4478-acaa-882e2d592fb8","Type":"ContainerStarted","Data":"d15200c382f03e5149013aea1d4f9dabcb26ffdcc4d8faaa5526d05a4e74aab6"}
Sep 09 20:12:13.001470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:13.001447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wqftt" event={"ID":"7b059ae5-251d-4a2b-a6e2-81571644fa39","Type":"ContainerStarted","Data":"bf42f42345a4bf6608cd640564a8a0e2a8b8d001f9ac4761101ab52b84644fcd"}
Sep 09 20:12:13.002633 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:13.002608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsvjt" event={"ID":"ffa78a3f-5d6e-4689-a436-65a2aafc1e01","Type":"ContainerStarted","Data":"31898f4a3e2f7a7b10154393d1dc9a68d101725f862a4b58b48047c5f6b0a4cd"}
Sep 09 20:12:14.849121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.848824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"]
Sep 09 20:12:14.870035 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.869874    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"]
Sep 09 20:12:14.870230 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.870082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:14.873710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.873224    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:12:14.873710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.873453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:14.875264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.874502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tswc9\""
Sep 09 20:12:14.875264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.874762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:12:14.875264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.874784    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:14.875264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.874970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:12:14.883434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.882357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"]
Sep 09 20:12:14.883434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.882409    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4p6zm"]
Sep 09 20:12:14.883434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.882870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:14.886691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.886013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:12:14.886691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.886190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 20:12:14.886691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.886222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bwtx5\""
Sep 09 20:12:14.886691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.886453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 20:12:14.906871 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.906751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"]
Sep 09 20:12:14.907006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.906895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:14.909710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.909686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:14.910001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.909984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm7xj\""
Sep 09 20:12:14.910262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.910246    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:14.910884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:14.910860    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:15.012878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.012841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsvjt" event={"ID":"ffa78a3f-5d6e-4689-a436-65a2aafc1e01","Type":"ContainerStarted","Data":"f9ced6f63d5419ad5296a8641e019ffe725442c12ed5bb5f5f9672728cb8b0f0"}
Sep 09 20:12:15.022858 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.022823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.023001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.022890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.022931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-wtmp\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.022957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-sys\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.022985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.023242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-textfile\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.023242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.023242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-accelerators-collector-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-tls\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dae08fee-6d37-4125-a77f-19e233dc3849-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-root\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-metrics-client-ca\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m49d\" (UniqueName: \"kubernetes.io/projected/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-kube-api-access-5m49d\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxhp\" (UniqueName: \"kubernetes.io/projected/dae08fee-6d37-4125-a77f-19e233dc3849-kube-api-access-mxxhp\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.023496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfdq\" (UniqueName: \"kubernetes.io/projected/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-api-access-9mfdq\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.023798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.023798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.023798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.023589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124126 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-wtmp\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-sys\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-textfile\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-accelerators-collector-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-tls\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dae08fee-6d37-4125-a77f-19e233dc3849-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.124514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-root\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-metrics-client-ca\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m49d\" (UniqueName: \"kubernetes.io/projected/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-kube-api-access-5m49d\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxhp\" (UniqueName: \"kubernetes.io/projected/dae08fee-6d37-4125-a77f-19e233dc3849-kube-api-access-mxxhp\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfdq\" (UniqueName: \"kubernetes.io/projected/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-api-access-9mfdq\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.124991 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-wtmp\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.124991 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.124845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-sys\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.125162 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.125124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.126124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.126101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dae08fee-6d37-4125-a77f-19e233dc3849-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.127306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.127281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.127772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.127749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-root\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.127849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.127802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-accelerators-collector-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.128483 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.128450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-metrics-client-ca\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.128593 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.128515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-textfile\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.131318 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.131292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.131533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.131511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.131946 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.131900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-node-exporter-tls\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.137056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.132655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.137056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.133406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.137056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.135397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfdq\" (UniqueName: \"kubernetes.io/projected/ed1ae4a7-c766-4d53-8f17-70ec080d2c71-kube-api-access-9mfdq\") pod \"kube-state-metrics-74cb57df7-xwmc2\" (UID: \"ed1ae4a7-c766-4d53-8f17-70ec080d2c71\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.137056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.135529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxhp\" (UniqueName: \"kubernetes.io/projected/dae08fee-6d37-4125-a77f-19e233dc3849-kube-api-access-mxxhp\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.137056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.137012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dae08fee-6d37-4125-a77f-19e233dc3849-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-hzblv\" (UID: \"dae08fee-6d37-4125-a77f-19e233dc3849\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.160797 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.154545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m49d\" (UniqueName: \"kubernetes.io/projected/9a5f94bb-01d4-4fdf-86e4-ec6eb126be78-kube-api-access-5m49d\") pod \"node-exporter-4p6zm\" (UID: \"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78\") " pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.187899 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.187866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"
Sep 09 20:12:15.209125 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.209020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"
Sep 09 20:12:15.219464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.219392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4p6zm"
Sep 09 20:12:15.286116 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:15.286063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5f94bb_01d4_4fdf_86e4_ec6eb126be78.slice/crio-4e3d744cc19677204eff72aadf049a21d177e0dd86fd779ca7bde41750c30375 WatchSource:0}: Error finding container 4e3d744cc19677204eff72aadf049a21d177e0dd86fd779ca7bde41750c30375: Status 404 returned error can't find the container with id 4e3d744cc19677204eff72aadf049a21d177e0dd86fd779ca7bde41750c30375
Sep 09 20:12:15.495207 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.495114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv"]
Sep 09 20:12:15.530223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:15.530149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2"]
Sep 09 20:12:16.019860 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:16.019814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4p6zm" event={"ID":"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78","Type":"ContainerStarted","Data":"4e3d744cc19677204eff72aadf049a21d177e0dd86fd779ca7bde41750c30375"}
Sep 09 20:12:16.022730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:16.022697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dsvjt" event={"ID":"ffa78a3f-5d6e-4689-a436-65a2aafc1e01","Type":"ContainerStarted","Data":"5b16ef85a4ff4a6ab75ebc8885ac02e5c608c2c8ae1715e6651b3b9ee6b27da4"}
Sep 09 20:12:16.040335 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:16.040283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dsvjt" podStartSLOduration=66.175323948 podStartE2EDuration="1m8.040265267s" totalImagesPullingTime="1.864941319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:16.038182288 +0000 UTC m=+68.874496751" watchObservedRunningTime="2026-09-09 20:12:16.040265267 +0000 UTC m=+68.876579733"
Sep 09 20:12:16.836887 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:16.836841    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae08fee_6d37_4125_a77f_19e233dc3849.slice/crio-6651756c1b5a3bb173689de9abe1ff98aead7790793f8667842e6aba7c79ebd2 WatchSource:0}: Error finding container 6651756c1b5a3bb173689de9abe1ff98aead7790793f8667842e6aba7c79ebd2: Status 404 returned error can't find the container with id 6651756c1b5a3bb173689de9abe1ff98aead7790793f8667842e6aba7c79ebd2
Sep 09 20:12:16.988259 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:16.988228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pn5rp"
Sep 09 20:12:17.032338 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:17.032302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv" event={"ID":"dae08fee-6d37-4125-a77f-19e233dc3849","Type":"ContainerStarted","Data":"6651756c1b5a3bb173689de9abe1ff98aead7790793f8667842e6aba7c79ebd2"}
Sep 09 20:12:17.808323 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:17.808282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1ae4a7_c766_4d53_8f17_70ec080d2c71.slice/crio-42c181fe84c0a301b304282efb97910b05173aed38472b9ec07a04b612d0cc30 WatchSource:0}: Error finding container 42c181fe84c0a301b304282efb97910b05173aed38472b9ec07a04b612d0cc30: Status 404 returned error can't find the container with id 42c181fe84c0a301b304282efb97910b05173aed38472b9ec07a04b612d0cc30
Sep 09 20:12:18.040853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.040546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2tw8s" event={"ID":"8517e6d9-209b-4478-acaa-882e2d592fb8","Type":"ContainerStarted","Data":"6c00835de3f6d452fe173561e220a7abc9191e199a2447a5f781873fcb771934"}
Sep 09 20:12:18.040853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.040667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:12:18.044336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.044300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wqftt" event={"ID":"7b059ae5-251d-4a2b-a6e2-81571644fa39","Type":"ContainerStarted","Data":"3d7ea67789de95467b3220a80c4c8481d520bb91ecf8cc066e6f57952d9fcb7e"}
Sep 09 20:12:18.046808 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.046700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv" event={"ID":"dae08fee-6d37-4125-a77f-19e233dc3849","Type":"ContainerStarted","Data":"5f1988da50761a09c711241aff7e1316c6f7a7427b780862de2e2f36fac202ae"}
Sep 09 20:12:18.046808 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.046773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv" event={"ID":"dae08fee-6d37-4125-a77f-19e233dc3849","Type":"ContainerStarted","Data":"e51b0ad704eb32d744f4317206224fa2ee92ae68b5dc92ca769198b843384382"}
Sep 09 20:12:18.048320 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.048281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2" event={"ID":"ed1ae4a7-c766-4d53-8f17-70ec080d2c71","Type":"ContainerStarted","Data":"42c181fe84c0a301b304282efb97910b05173aed38472b9ec07a04b612d0cc30"}
Sep 09 20:12:18.055322 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.055246    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2tw8s" podStartSLOduration=65.18180594 podStartE2EDuration="1m10.055226221s" totalImagesPullingTime="4.873420281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:18.054413731 +0000 UTC m=+70.890728195" watchObservedRunningTime="2026-09-09 20:12:18.055226221 +0000 UTC m=+70.891540685"
Sep 09 20:12:18.068394 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:18.068211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wqftt" podStartSLOduration=65.002597058 podStartE2EDuration="1m10.068193844s" totalImagesPullingTime="5.065596786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:18.067851263 +0000 UTC m=+70.904165738" watchObservedRunningTime="2026-09-09 20:12:18.068193844 +0000 UTC m=+70.904508361"
Sep 09 20:12:19.055461 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.055420    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a5f94bb-01d4-4fdf-86e4-ec6eb126be78" containerID="cca8cb38842a419d0b091ce15d3d2024d20afa2f43198d07c1143fc50e6f258a" exitCode=0
Sep 09 20:12:19.055930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.055519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4p6zm" event={"ID":"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78","Type":"ContainerDied","Data":"cca8cb38842a419d0b091ce15d3d2024d20afa2f43198d07c1143fc50e6f258a"}
Sep 09 20:12:19.618804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.618777    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"]
Sep 09 20:12:19.623018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.623000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:19.625409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.625383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 20:12:19.625527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.625514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7tf9w\""
Sep 09 20:12:19.629112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.629004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"]
Sep 09 20:12:19.655070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.655046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjmv\" (UniqueName: \"kubernetes.io/projected/8ae15eb3-7466-40fb-9665-54a46938bc15-kube-api-access-bjjmv\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:19.655229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.655079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:19.757634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.756478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjmv\" (UniqueName: \"kubernetes.io/projected/8ae15eb3-7466-40fb-9665-54a46938bc15-kube-api-access-bjjmv\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:19.757634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.756599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:19.757634 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:12:19.756886    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 20:12:19.757634 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:12:19.756959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert podName:8ae15eb3-7466-40fb-9665-54a46938bc15 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:20.256935381 +0000 UTC m=+73.093249846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-vlc44" (UID: "8ae15eb3-7466-40fb-9665-54a46938bc15") : secret "monitoring-plugin-cert" not found
Sep 09 20:12:19.768789 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:19.768047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjmv\" (UniqueName: \"kubernetes.io/projected/8ae15eb3-7466-40fb-9665-54a46938bc15-kube-api-access-bjjmv\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:20.061996 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.061892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv" event={"ID":"dae08fee-6d37-4125-a77f-19e233dc3849","Type":"ContainerStarted","Data":"8c1fb77ed55fef4c8bc60f63e799790c99e25e2ed47a27e2a80668982e578135"}
Sep 09 20:12:20.064838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.064795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2" event={"ID":"ed1ae4a7-c766-4d53-8f17-70ec080d2c71","Type":"ContainerStarted","Data":"92f6308d5daa9b822ed85bda8d820d90a2fcc322da08e70017030179877e4f95"}
Sep 09 20:12:20.064986 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.064875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2" event={"ID":"ed1ae4a7-c766-4d53-8f17-70ec080d2c71","Type":"ContainerStarted","Data":"67c2ef05b7dbeed68c203eecef7a2da6f487bf1cb683ab3b47e7d531ed43506e"}
Sep 09 20:12:20.064986 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.064909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2" event={"ID":"ed1ae4a7-c766-4d53-8f17-70ec080d2c71","Type":"ContainerStarted","Data":"0169e501cdf74a2a464fc8811a42419fe08cd496fe429b095ae686f7ec466897"}
Sep 09 20:12:20.067289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.067267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4p6zm" event={"ID":"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78","Type":"ContainerStarted","Data":"a7f5b9ccc33c18a2f2ce375729399103af5c824f74ae19fa7c83a4f8650ba8f5"}
Sep 09 20:12:20.067401 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.067305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4p6zm" event={"ID":"9a5f94bb-01d4-4fdf-86e4-ec6eb126be78","Type":"ContainerStarted","Data":"da127b83be428917f6f9a6f1ee21252eb3783682debb07a647008c91741dba1d"}
Sep 09 20:12:20.073889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.073863    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"]
Sep 09 20:12:20.077120 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.077104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.079822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.079717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 20:12:20.079934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.079828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 20:12:20.079934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.079902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 20:12:20.079934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.079907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 20:12:20.080128 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.079974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hdcl8\""
Sep 09 20:12:20.080287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.080265    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 20:12:20.083132 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.083078    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-hzblv" podStartSLOduration=4.544852077 podStartE2EDuration="6.083061077s" totalImagesPullingTime="1.538209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.081468951 +0000 UTC m=+72.917783415" watchObservedRunningTime="2026-09-09 20:12:20.083061077 +0000 UTC m=+72.919375540"
Sep 09 20:12:20.085418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.085394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 20:12:20.090514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.090446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"]
Sep 09 20:12:20.121708 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.121655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-xwmc2" podStartSLOduration=4.414569647 podStartE2EDuration="6.121638684s" totalImagesPullingTime="1.707069037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.120980507 +0000 UTC m=+72.957294973" watchObservedRunningTime="2026-09-09 20:12:20.121638684 +0000 UTC m=+72.957953147"
Sep 09 20:12:20.139337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.139281    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4p6zm" podStartSLOduration=2.931993594 podStartE2EDuration="6.139263671s" totalImagesPullingTime="3.134636769s" totalInitContainerRuntime="72.633308ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.138337433 +0000 UTC m=+72.974651896" watchObservedRunningTime="2026-09-09 20:12:20.139263671 +0000 UTC m=+72.975578134"
Sep 09 20:12:20.158884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.158845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.158993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.158971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-federate-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-serving-certs-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-metrics-client-ca\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.159649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.159373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckfm\" (UniqueName: \"kubernetes.io/projected/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-kube-api-access-zckfm\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-federate-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260530 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260530 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:20.260589 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260589 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-serving-certs-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-metrics-client-ca\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.260756 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.260676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckfm\" (UniqueName: \"kubernetes.io/projected/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-kube-api-access-zckfm\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.261725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.261699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-metrics-client-ca\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.261947 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.261909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-serving-certs-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.262863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.262805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.263342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.263319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.263497 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.263473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-secret-telemeter-client\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.263617 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.263595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8ae15eb3-7466-40fb-9665-54a46938bc15-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-vlc44\" (UID: \"8ae15eb3-7466-40fb-9665-54a46938bc15\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:20.263743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.263630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-federate-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.263800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.263777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-telemeter-client-tls\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.268453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.268431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckfm\" (UniqueName: \"kubernetes.io/projected/8a4bb0f4-5fa8-4554-aa77-af443b5f74a8-kube-api-access-zckfm\") pod \"telemeter-client-56fcf65b95-hvjv4\" (UID: \"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8\") " pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.388401 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.388312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"
Sep 09 20:12:20.542769 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.542730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:20.662873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.662850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56fcf65b95-hvjv4"]
Sep 09 20:12:20.664715 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:20.664689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4bb0f4_5fa8_4554_aa77_af443b5f74a8.slice/crio-e2aa4106017d3197b30cc9c180eeb88db498eafb82cee599bcf26a3235461d18 WatchSource:0}: Error finding container e2aa4106017d3197b30cc9c180eeb88db498eafb82cee599bcf26a3235461d18: Status 404 returned error can't find the container with id e2aa4106017d3197b30cc9c180eeb88db498eafb82cee599bcf26a3235461d18
Sep 09 20:12:20.812951 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:20.812926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"]
Sep 09 20:12:20.815354 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:20.815327    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae15eb3_7466_40fb_9665_54a46938bc15.slice/crio-fd7d3d9d615227478df738056e96894cf1e42f866c38b42921df7fed056e0b1d WatchSource:0}: Error finding container fd7d3d9d615227478df738056e96894cf1e42f866c38b42921df7fed056e0b1d: Status 404 returned error can't find the container with id fd7d3d9d615227478df738056e96894cf1e42f866c38b42921df7fed056e0b1d
Sep 09 20:12:21.072171 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.072065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44" event={"ID":"8ae15eb3-7466-40fb-9665-54a46938bc15","Type":"ContainerStarted","Data":"fd7d3d9d615227478df738056e96894cf1e42f866c38b42921df7fed056e0b1d"}
Sep 09 20:12:21.073426 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.073395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4" event={"ID":"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8","Type":"ContainerStarted","Data":"e2aa4106017d3197b30cc9c180eeb88db498eafb82cee599bcf26a3235461d18"}
Sep 09 20:12:21.117663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.117627    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:12:21.124820 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.124786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.127583 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:12:21.127742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:12:21.127807 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127782    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:12:21.127870 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:12:21.128006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127983    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:12:21.128006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.127997    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:12:21.128181 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:12:21.128277 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128255    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:12:21.128277 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:12:21.128524 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:12:21.128634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fkgm3a6mho1al\""
Sep 09 20:12:21.128768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:12:21.129005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.128991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:12:21.129005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.129000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:12:21.129142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.129017    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:12:21.129142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.129000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hzmqp\""
Sep 09 20:12:21.131464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.131440    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:12:21.138287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.138265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:12:21.168419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.168988 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.168967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkm8\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169711 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169711 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.169711 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.169575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.270866 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.270826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.270866 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.270866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.270884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.270909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.270929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkm8\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.271689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.272152 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.271877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.273607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.272461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.273607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.272660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.273607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.273023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.273607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.273059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.274225 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.274169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.274547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.274516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.275514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.274720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.275514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.275456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.275965 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.275867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.275965 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.275924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.276163 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.276139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.276326 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.276304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.276744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.276720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.277246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.277221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.277377 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.277356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.277489 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.277468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.277745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.277727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.278102 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.278069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.279687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.279664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkm8\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8\") pod \"prometheus-k8s-0\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.435189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.435157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:21.725733 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:21.725636    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:12:21.730847 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:12:21.730813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef5d5ca_eefb_48f7_b974_11159cfacac7.slice/crio-21fdf52ff5c252cb89acb77d70f174f0f4ec43c744207acd125cf8f229b3f880 WatchSource:0}: Error finding container 21fdf52ff5c252cb89acb77d70f174f0f4ec43c744207acd125cf8f229b3f880: Status 404 returned error can't find the container with id 21fdf52ff5c252cb89acb77d70f174f0f4ec43c744207acd125cf8f229b3f880
Sep 09 20:12:22.079294 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:22.079213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"21fdf52ff5c252cb89acb77d70f174f0f4ec43c744207acd125cf8f229b3f880"}
Sep 09 20:12:23.085078 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.084971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4" event={"ID":"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8","Type":"ContainerStarted","Data":"32f4918bd0becba4734c9027771b9d19e4961ee95385804a8241b5f2b6e534ce"}
Sep 09 20:12:23.086703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.086671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44" event={"ID":"8ae15eb3-7466-40fb-9665-54a46938bc15","Type":"ContainerStarted","Data":"bb3016cdb881ee459bd17ca09b1a033c41bbeedcffe1427c2b04ddb6f92472a0"}
Sep 09 20:12:23.086965 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.086944    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:23.092921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.092897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44"
Sep 09 20:12:23.104897 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.104840    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-vlc44" podStartSLOduration=2.261140602 podStartE2EDuration="4.1048191s" totalImagesPullingTime="1.843678498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:23.103273437 +0000 UTC m=+75.939587900" watchObservedRunningTime="2026-09-09 20:12:23.1048191 +0000 UTC m=+75.941133566"
Sep 09 20:12:23.274825 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.274784    2579 patch_prober.go:28] interesting pod/image-registry-97fdbc685-p5s4b container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:12:23.275012 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:23.274852    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b" podUID="121425d6-de91-426a-bcee-c44d2adc5f24" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:24.092967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.092934    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5" exitCode=0
Sep 09 20:12:24.093471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.093026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"}
Sep 09 20:12:24.095567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.095530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4" event={"ID":"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8","Type":"ContainerStarted","Data":"bcd5e960786fe903955d61858c1a93bf4c25cd4318180a50c657aae85ca6123b"}
Sep 09 20:12:24.095662 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.095607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4" event={"ID":"8a4bb0f4-5fa8-4554-aa77-af443b5f74a8","Type":"ContainerStarted","Data":"d9d23fb2f7b8a78e231da7308a81f3e1fba9715b231bbb0b9199cb0ecfafbf1c"}
Sep 09 20:12:24.140132 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.140058    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56fcf65b95-hvjv4" podStartSLOduration=1.4745891119999999 podStartE2EDuration="4.140041124s" totalImagesPullingTime="2.665452012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:24.138728248 +0000 UTC m=+76.975042734" watchObservedRunningTime="2026-09-09 20:12:24.140041124 +0000 UTC m=+76.976355586"
Sep 09 20:12:24.973595 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:24.973562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97fdbc685-p5s4b"
Sep 09 20:12:28.117960 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:28.117914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"}
Sep 09 20:12:28.118398 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:28.117999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"}
Sep 09 20:12:29.125220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:29.125190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"}
Sep 09 20:12:30.132570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:30.132537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"}
Sep 09 20:12:30.132944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:30.132600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"}
Sep 09 20:12:30.132944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:30.132627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerStarted","Data":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"}
Sep 09 20:12:30.162428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:30.162375    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.797402858 podStartE2EDuration="9.162357514s" totalImagesPullingTime="6.345649473s" totalInitContainerRuntime="19.305183ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:30.159916664 +0000 UTC m=+82.996231151" watchObservedRunningTime="2026-09-09 20:12:30.162357514 +0000 UTC m=+82.998671976"
Sep 09 20:12:31.436213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:31.436167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:12:49.058891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:12:49.058856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2tw8s"
Sep 09 20:13:21.435843 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:21.435800    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:21.455634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:21.455594    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:22.338851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:22.338819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:39.401279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401242    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401535    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="prometheus" containerID="cri-o://02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc" gracePeriod=600
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401567    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-web" containerID="cri-o://1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e" gracePeriod=600
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401606    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="thanos-sidecar" containerID="cri-o://f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc" gracePeriod=600
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401567    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989" gracePeriod=600
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401616    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy" containerID="cri-o://246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd" gracePeriod=600
Sep 09 20:13:39.401700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.401644    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="config-reloader" containerID="cri-o://110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49" gracePeriod=600
Sep 09 20:13:39.633275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.633247    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:39.731273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731606    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.731703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qkm8\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.731703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731822    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.731874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.732065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db\") pod \"bef5d5ca-eefb-48f7-b974-11159cfacac7\" (UID: \"bef5d5ca-eefb-48f7-b974-11159cfacac7\") "
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732484    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732587    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732605    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.732643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732620    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.732970 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.732941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.733994 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.733664    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:13:39.734553 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.734553 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.734712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.734712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734616    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config" (OuterVolumeSpecName: "config") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.734827 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.735032 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.734978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.735767 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.735730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.736221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.736194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:39.736384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.736363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.736832 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.736801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.737036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.737018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:39.737180 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.737163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8" (OuterVolumeSpecName: "kube-api-access-2qkm8") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "kube-api-access-2qkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:39.737267 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.737245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out" (OuterVolumeSpecName: "config-out") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:13:39.746459 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.746436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config" (OuterVolumeSpecName: "web-config") pod "bef5d5ca-eefb-48f7-b974-11159cfacac7" (UID: "bef5d5ca-eefb-48f7-b974-11159cfacac7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:39.833602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833562    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833594    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-tls-assets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833605    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-web-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833612    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-config-out\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833621    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833631    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833641    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-kube-rbac-proxy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833651    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-grpc-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833658    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bef5d5ca-eefb-48f7-b974-11159cfacac7-prometheus-k8s-db\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833665    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-metrics-client-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833673    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833683    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833691    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef5d5ca-eefb-48f7-b974-11159cfacac7-configmap-metrics-client-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833699    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833707    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bef5d5ca-eefb-48f7-b974-11159cfacac7-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:39.833831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:39.833715    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkm8\" (UniqueName: \"kubernetes.io/projected/bef5d5ca-eefb-48f7-b974-11159cfacac7-kube-api-access-2qkm8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:13:40.392636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392597    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989" exitCode=0
Sep 09 20:13:40.392636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392628    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd" exitCode=0
Sep 09 20:13:40.392636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392637    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e" exitCode=0
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392646    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc" exitCode=0
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392654    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49" exitCode=0
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392661    2579 generic.go:356] "Generic (PLEG): container finished" podID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc" exitCode=0
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"}
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"}
Sep 09 20:13:40.392864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"}
Sep 09 20:13:40.393112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"}
Sep 09 20:13:40.393112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392906    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.393112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"}
Sep 09 20:13:40.393112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"}
Sep 09 20:13:40.393112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.392962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bef5d5ca-eefb-48f7-b974-11159cfacac7","Type":"ContainerDied","Data":"21fdf52ff5c252cb89acb77d70f174f0f4ec43c744207acd125cf8f229b3f880"}
Sep 09 20:13:40.401170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.401147    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.408696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.408675    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.416738 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.416681    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.417208 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.417184    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:40.421431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.421407    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:40.435648 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.435616    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.443336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.443317    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.445810 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.445787    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:40.446042 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446031    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="init-config-reloader"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446044    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="init-config-reloader"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446051    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-web"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-web"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446066    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="config-reloader"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446072    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="config-reloader"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446104    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="thanos-sidecar"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446113    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="thanos-sidecar"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446119    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy"
Sep 09 20:13:40.446082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446124    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="prometheus"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="prometheus"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446145    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-thanos"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-thanos"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="config-reloader"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446203    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="thanos-sidecar"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446209    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-thanos"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446215    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="prometheus"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446220    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy-web"
Sep 09 20:13:40.446458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.446224    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" containerName="kube-rbac-proxy"
Sep 09 20:13:40.451144 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.451125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.452045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.452025    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.454357 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454335    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:13:40.454455 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454398    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:13:40.454455 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454429    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:13:40.454558 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:13:40.454558 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454429    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fkgm3a6mho1al\""
Sep 09 20:13:40.454558 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:13:40.454704 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:13:40.454758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:13:40.454808 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:13:40.454860 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:13:40.454860 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.454842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:13:40.455315 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.455295    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hzmqp\""
Sep 09 20:13:40.455411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.455330    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:13:40.455634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.455531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:13:40.455634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.455542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:13:40.458522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.458502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:13:40.459210 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.459193    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:13:40.462888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.462796    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.464675 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.464551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.464675 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.464590    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.464675 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.464634    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.465608 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.465584    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.465749 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.465617    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.465749 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.465644    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.466037 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.466016    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.466117 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.466117 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466059    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.466349 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.466334    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.466392 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.466392 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466366    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.466508 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:40.466665 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.466648    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.466710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.466710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466683    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.466946 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.466928    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.467009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466956    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.467009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.466977    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.467314 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:13:40.467298    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.467355 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467319    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.467355 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467335    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.467584 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467565    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.467630 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467585    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.467827 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467807    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.467894 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.467828    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.468151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468080    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.468151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468152    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.468479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468455    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.468479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468479    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.468749 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468729    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.468808 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.468750    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.469033 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469013    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.469112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469035    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.469293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469277    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.469293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469292    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.469576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469550    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.469576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469575    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.469828 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469808    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.469828 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.469828    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.470126 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470080    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.470126 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470127    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.470398 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470377    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.470449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470398    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.470653 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470634    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.470727 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470656    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.470921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470897    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.470921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.470920    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.471223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471206    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.471279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471223    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.471488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.471546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471488    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.471728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471709    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.471794 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471730    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.471969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471953    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.472029 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.471971    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.472219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472202    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.472298 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472220    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.472450 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472422    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.472450 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472443    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.472699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.472743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472703    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.472959 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.473017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.472958    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.473192 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.473192 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473192    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.473453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473428    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.473507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473455    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.473733 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473716    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.473776 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473734    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.473976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.474030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.473977    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.474254 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474235    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.474300 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474254    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.474481 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474465    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.474522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474482    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.474718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474701    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.474718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474717    2579 scope.go:117] "RemoveContainer" containerID="325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"
Sep 09 20:13:40.474953 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989"} err="failed to get container status \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": rpc error: code = NotFound desc = could not find container \"325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989\": container with ID starting with 325add836f0823e27a89527db289b256ec8775a32bb19247294afa7e07a1d989 not found: ID does not exist"
Sep 09 20:13:40.474998 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.474953    2579 scope.go:117] "RemoveContainer" containerID="246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"
Sep 09 20:13:40.475208 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475190    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd"} err="failed to get container status \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": rpc error: code = NotFound desc = could not find container \"246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd\": container with ID starting with 246588870f037455437a2ea386946c632af2610f0e9e2f1b5a3a3f0cf8c45bbd not found: ID does not exist"
Sep 09 20:13:40.475208 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475208    2579 scope.go:117] "RemoveContainer" containerID="1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"
Sep 09 20:13:40.475429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475414    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e"} err="failed to get container status \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": rpc error: code = NotFound desc = could not find container \"1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e\": container with ID starting with 1779d4d5fd3e27a80fff4157bd204a8a7f0ecf6b076e115239ac399926b2533e not found: ID does not exist"
Sep 09 20:13:40.475478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475429    2579 scope.go:117] "RemoveContainer" containerID="f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"
Sep 09 20:13:40.475699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475684    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc"} err="failed to get container status \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": rpc error: code = NotFound desc = could not find container \"f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc\": container with ID starting with f13e6db357e8593138782be039c806c4e01016c541b37063603588668d54a0fc not found: ID does not exist"
Sep 09 20:13:40.475755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475699    2579 scope.go:117] "RemoveContainer" containerID="110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"
Sep 09 20:13:40.475930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475915    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49"} err="failed to get container status \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": rpc error: code = NotFound desc = could not find container \"110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49\": container with ID starting with 110e531a7de938daffc809daf899ec120f9859d81627a63487d0544f8ed14c49 not found: ID does not exist"
Sep 09 20:13:40.475969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.475929    2579 scope.go:117] "RemoveContainer" containerID="02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"
Sep 09 20:13:40.476231 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.476202    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc"} err="failed to get container status \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": rpc error: code = NotFound desc = could not find container \"02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc\": container with ID starting with 02e0d1a9550ba95c82dc44e0014e79b35ac7e215fef7aeb05d332c92df052cdc not found: ID does not exist"
Sep 09 20:13:40.476231 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.476231    2579 scope.go:117] "RemoveContainer" containerID="4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"
Sep 09 20:13:40.476509 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.476492    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5"} err="failed to get container status \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": rpc error: code = NotFound desc = could not find container \"4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5\": container with ID starting with 4529c90a16b42528ac320add65fdb922b5098238369e6cfe153240c23b4405c5 not found: ID does not exist"
Sep 09 20:13:40.540299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-config-out\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540537 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27q4k\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-kube-api-access-27q4k\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-web-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.540835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.540831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-config-out\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.641816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27q4k\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-kube-api-access-27q4k\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.641981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-web-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.642803 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.642751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.644493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.643379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.644493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.643508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.644493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.643695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.644493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.644380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.644935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.644976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.645168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.645672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.645825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.645938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.645875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.646564 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.646516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/785c8670-0a48-4255-9a8b-a80764a3eed9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.647336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.647307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-web-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.647424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.647307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.647494 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.647471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.648374 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.648355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-config\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.648429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.648407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/785c8670-0a48-4255-9a8b-a80764a3eed9-config-out\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.648612 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.648590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/785c8670-0a48-4255-9a8b-a80764a3eed9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.651386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.651367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27q4k\" (UniqueName: \"kubernetes.io/projected/785c8670-0a48-4255-9a8b-a80764a3eed9-kube-api-access-27q4k\") pod \"prometheus-k8s-0\" (UID: \"785c8670-0a48-4255-9a8b-a80764a3eed9\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:40.765804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:40.765745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:13:41.047234 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:41.047207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:13:41.048677 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:13:41.048654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785c8670_0a48_4255_9a8b_a80764a3eed9.slice/crio-a27764c1c99072c351aa88de06f62551cf83d7ebb2b5cbb6aa1642f473880879 WatchSource:0}: Error finding container a27764c1c99072c351aa88de06f62551cf83d7ebb2b5cbb6aa1642f473880879: Status 404 returned error can't find the container with id a27764c1c99072c351aa88de06f62551cf83d7ebb2b5cbb6aa1642f473880879
Sep 09 20:13:41.398800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:41.398758    2579 generic.go:356] "Generic (PLEG): container finished" podID="785c8670-0a48-4255-9a8b-a80764a3eed9" containerID="cff020444b54c54d2b4241cce9965a10be86eb72431c2ad1912160be788f8139" exitCode=0
Sep 09 20:13:41.398980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:41.398852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerDied","Data":"cff020444b54c54d2b4241cce9965a10be86eb72431c2ad1912160be788f8139"}
Sep 09 20:13:41.398980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:41.398968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"a27764c1c99072c351aa88de06f62551cf83d7ebb2b5cbb6aa1642f473880879"}
Sep 09 20:13:41.615772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:41.615726    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef5d5ca-eefb-48f7-b974-11159cfacac7" path="/var/lib/kubelet/pods/bef5d5ca-eefb-48f7-b974-11159cfacac7/volumes"
Sep 09 20:13:42.407546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"49aaff1e3e915bb5cf4fe024c8755ed2ab839c94212fcdc19b56e7cd0875e0d2"}
Sep 09 20:13:42.407725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"2d46c3c8d726cdecddc0282a0924e55957204ff533eb142a0558dfcc87e2420e"}
Sep 09 20:13:42.407725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"10e0a545b4bca03ac01bd5bb990241613e07f56b91523dda514307ba2213250f"}
Sep 09 20:13:42.407725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"b9dc8e7fd4d0b4a391f90ae3115fe57aadf53a9ceaa4d87f6fda57d1aade1654"}
Sep 09 20:13:42.407725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"a0b0f163aceb78b17b97572c1d844e341d6fbe17c8f8a876a164d380490a04a3"}
Sep 09 20:13:42.407725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.407679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"785c8670-0a48-4255-9a8b-a80764a3eed9","Type":"ContainerStarted","Data":"0576d488af5a0014f2c25fbf8f4e36a93591c42a29e04d59db46c513e59bc7eb"}
Sep 09 20:13:42.443275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:42.443199    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.422325711 podStartE2EDuration="2.443080143s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.754432ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:42.435482787 +0000 UTC m=+155.271797250" watchObservedRunningTime="2026-09-09 20:13:42.443080143 +0000 UTC m=+155.279394607"
Sep 09 20:13:45.765950 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:13:45.765906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:14:40.766307 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:14:40.766266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:14:40.783265 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:14:40.783234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:14:41.642990 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:14:41.642958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:16:07.529365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:16:07.529332    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:17:50.378615 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.378573    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-rjhx6"]
Sep 09 20:17:50.381954 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.381930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:50.384477 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.384455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z7jn8\""
Sep 09 20:17:50.384614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.384496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:17:50.385497 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.385477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:17:50.385594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.385534    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:17:50.391948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.391927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-rjhx6"]
Sep 09 20:17:50.454779 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.454736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg4f\" (UniqueName: \"kubernetes.io/projected/c51e2be1-75c5-4565-8c39-77250feceb42-kube-api-access-6wg4f\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:50.454959 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.454824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:50.555225 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.555189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:50.555365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.555251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg4f\" (UniqueName: \"kubernetes.io/projected/c51e2be1-75c5-4565-8c39-77250feceb42-kube-api-access-6wg4f\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:50.555365 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:17:50.555345    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:17:50.555432 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:17:50.555426    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs podName:c51e2be1-75c5-4565-8c39-77250feceb42 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:51.055406818 +0000 UTC m=+403.891721259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs") pod "model-serving-api-7b954ffc95-rjhx6" (UID: "c51e2be1-75c5-4565-8c39-77250feceb42") : secret "model-serving-api-tls" not found
Sep 09 20:17:50.563643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:50.563609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg4f\" (UniqueName: \"kubernetes.io/projected/c51e2be1-75c5-4565-8c39-77250feceb42-kube-api-access-6wg4f\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:51.059363 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:51.059320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:51.061912 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:51.061893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51e2be1-75c5-4565-8c39-77250feceb42-tls-certs\") pod \"model-serving-api-7b954ffc95-rjhx6\" (UID: \"c51e2be1-75c5-4565-8c39-77250feceb42\") " pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:51.293566 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:51.293519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:51.563853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:51.563824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-rjhx6"]
Sep 09 20:17:51.566233 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:17:51.566207    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51e2be1_75c5_4565_8c39_77250feceb42.slice/crio-113b705d7db2e772a0ffeb1d4b22123aee89e786c202739312c1e7a1fa87d7d6 WatchSource:0}: Error finding container 113b705d7db2e772a0ffeb1d4b22123aee89e786c202739312c1e7a1fa87d7d6: Status 404 returned error can't find the container with id 113b705d7db2e772a0ffeb1d4b22123aee89e786c202739312c1e7a1fa87d7d6
Sep 09 20:17:51.568365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:51.568346    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:17:52.302158 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:52.302121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-rjhx6" event={"ID":"c51e2be1-75c5-4565-8c39-77250feceb42","Type":"ContainerStarted","Data":"113b705d7db2e772a0ffeb1d4b22123aee89e786c202739312c1e7a1fa87d7d6"}
Sep 09 20:17:54.311725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:54.311689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-rjhx6" event={"ID":"c51e2be1-75c5-4565-8c39-77250feceb42","Type":"ContainerStarted","Data":"e676e56041175471fb10117712692b013625487cb1bf9dfd1fb8140768fb00d4"}
Sep 09 20:17:54.312138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:54.311808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:17:54.328427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:17:54.328370    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-rjhx6" podStartSLOduration=1.8077059389999999 podStartE2EDuration="4.328355838s" totalImagesPullingTime="2.520649899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:54.326956637 +0000 UTC m=+407.163271100" watchObservedRunningTime="2026-09-09 20:17:54.328355838 +0000 UTC m=+407.164670301"
Sep 09 20:18:05.319272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:05.319237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-rjhx6"
Sep 09 20:18:07.091385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.091349    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-frblp"]
Sep 09 20:18:07.098033 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.098014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:18:07.100649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.100623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:18:07.100942 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.100925    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:18:07.101018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.100925    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rm66\""
Sep 09 20:18:07.101559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.101534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-frblp"]
Sep 09 20:18:07.193202 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.193157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtwz\" (UniqueName: \"kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.193202 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.193201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.294492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.294460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtwz\" (UniqueName: \"kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.294492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.294498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.294826 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.294812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.302740 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.302716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtwz\" (UniqueName: \"kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz\") pod \"s3-init-frblp\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") " pod="kserve/s3-init-frblp"
Sep 09 20:18:07.413462 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.413430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:18:07.682481 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:07.682393    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-frblp"]
Sep 09 20:18:07.686016 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:18:07.685989    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1150fc2e_5ef1_4b79_9fe1_f4c3c262deb7.slice/crio-06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22 WatchSource:0}: Error finding container 06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22: Status 404 returned error can't find the container with id 06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22
Sep 09 20:18:08.364028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:08.363985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerStarted","Data":"06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22"}
Sep 09 20:18:11.269569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:11.269549    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:18:11.377804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:11.377764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerStarted","Data":"36c4e2b16eceb554f49bbd1cec126daaf30c4635d495383b6926f73a2036aa0a"}
Sep 09 20:18:13.387843 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:13.387760    2579 generic.go:356] "Generic (PLEG): container finished" podID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerID="36c4e2b16eceb554f49bbd1cec126daaf30c4635d495383b6926f73a2036aa0a" exitCode=0
Sep 09 20:18:13.387843 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:13.387835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerDied","Data":"36c4e2b16eceb554f49bbd1cec126daaf30c4635d495383b6926f73a2036aa0a"}
Sep 09 20:18:18.068752 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:18.068728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:18:18.414501 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:18.414463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerStarted","Data":"685e7ed6ad6332ad63702156f04ac3442559d7f6d8ff0d27a23a0648d52b4bc1"}
Sep 09 20:18:18.430664 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:18:18.430612    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-frblp" podStartSLOduration=1.450346797 podStartE2EDuration="11.43059583s" totalImagesPullingTime="8.250830563s" totalInitContainerRuntime="1.72941847s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:18.429238425 +0000 UTC m=+431.265552900" watchObservedRunningTime="2026-09-09 20:18:18.43059583 +0000 UTC m=+431.266910293"
Sep 09 20:19:23.606188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:23.606042    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/s3-init-frblp" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="s3-init" containerID="cri-o://685e7ed6ad6332ad63702156f04ac3442559d7f6d8ff0d27a23a0648d52b4bc1" gracePeriod=30
Sep 09 20:19:23.656889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:23.656851    2579 generic.go:356] "Generic (PLEG): container finished" podID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerID="685e7ed6ad6332ad63702156f04ac3442559d7f6d8ff0d27a23a0648d52b4bc1" exitCode=0
Sep 09 20:19:23.657038 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:23.656925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerDied","Data":"685e7ed6ad6332ad63702156f04ac3442559d7f6d8ff0d27a23a0648d52b4bc1"}
Sep 09 20:19:24.662275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:24.662244    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:19:24.662275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:24.662267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-frblp" event={"ID":"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7","Type":"ContainerDied","Data":"06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22"}
Sep 09 20:19:24.662693 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:24.662322    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06edfe295152d5b3724ef4a928632be32435e932e7f42ad644142b778b9fed22"
Sep 09 20:19:25.665334 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.665302    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:19:25.673360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.673333    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:19:25.758510 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.758466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rtwz\" (UniqueName: \"kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz\") pod \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") "
Sep 09 20:19:25.758706 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.758576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache\") pod \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\" (UID: \"1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7\") "
Sep 09 20:19:25.760909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.760882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz" (OuterVolumeSpecName: "kube-api-access-7rtwz") pod "1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" (UID: "1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7"). InnerVolumeSpecName "kube-api-access-7rtwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:25.777621 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.777579    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" (UID: "1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:25.859444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.859407    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:25.859444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:25.859436    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtwz\" (UniqueName: \"kubernetes.io/projected/1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7-kube-api-access-7rtwz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:26.668969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:26.668939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-frblp"
Sep 09 20:19:28.896789 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.896750    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:28.897223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.897056    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="s3-init"
Sep 09 20:19:28.897223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.897067    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="s3-init"
Sep 09 20:19:28.897223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.897113    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="download-hf-model"
Sep 09 20:19:28.897223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.897119    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="download-hf-model"
Sep 09 20:19:28.897223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.897161    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7" containerName="s3-init"
Sep 09 20:19:28.900250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.900234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:28.902755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.902729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 20:19:28.902881 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.902783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rm66\""
Sep 09 20:19:28.907390 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.907368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:28.981807 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.981761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:28.982042 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:28.981868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntb98\" (UniqueName: \"kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.082914 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.082865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntb98\" (UniqueName: \"kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.083127 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.082954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.083364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.083345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.091115 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.091052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntb98\" (UniqueName: \"kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98\") pod \"seaweedfs-tls-custom-6644bf5d6c-s46gt\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.210224 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.210133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:29.481921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.481892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:29.483472 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:19:29.483441    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24123dc_801b_41ea_953f_1adc49100354.slice/crio-bb10055675520987ce30f1d12be23c1d0ab5842171e1a5cecb5cd511d9346d80 WatchSource:0}: Error finding container bb10055675520987ce30f1d12be23c1d0ab5842171e1a5cecb5cd511d9346d80: Status 404 returned error can't find the container with id bb10055675520987ce30f1d12be23c1d0ab5842171e1a5cecb5cd511d9346d80
Sep 09 20:19:29.682411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:29.682365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" event={"ID":"d24123dc-801b-41ea-953f-1adc49100354","Type":"ContainerStarted","Data":"bb10055675520987ce30f1d12be23c1d0ab5842171e1a5cecb5cd511d9346d80"}
Sep 09 20:19:32.694974 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:32.694886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" event={"ID":"d24123dc-801b-41ea-953f-1adc49100354","Type":"ContainerStarted","Data":"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"}
Sep 09 20:19:32.710482 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:32.710426    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" podStartSLOduration=1.755328146 podStartE2EDuration="4.710411417s" totalImagesPullingTime="2.955083271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:32.709181498 +0000 UTC m=+505.545495962" watchObservedRunningTime="2026-09-09 20:19:32.710411417 +0000 UTC m=+505.546725881"
Sep 09 20:19:34.507874 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:34.507834    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:34.703424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:34.703380    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" podUID="d24123dc-801b-41ea-953f-1adc49100354" containerName="seaweedfs-tls-custom" containerID="cri-o://a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625" gracePeriod=30
Sep 09 20:19:35.941354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:35.941328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:36.045354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.045237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntb98\" (UniqueName: \"kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98\") pod \"d24123dc-801b-41ea-953f-1adc49100354\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") "
Sep 09 20:19:36.045354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.045312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data\") pod \"d24123dc-801b-41ea-953f-1adc49100354\" (UID: \"d24123dc-801b-41ea-953f-1adc49100354\") "
Sep 09 20:19:36.046243 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.046210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data" (OuterVolumeSpecName: "data") pod "d24123dc-801b-41ea-953f-1adc49100354" (UID: "d24123dc-801b-41ea-953f-1adc49100354"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:36.047527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.047506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98" (OuterVolumeSpecName: "kube-api-access-ntb98") pod "d24123dc-801b-41ea-953f-1adc49100354" (UID: "d24123dc-801b-41ea-953f-1adc49100354"). InnerVolumeSpecName "kube-api-access-ntb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:36.146419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.146377    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntb98\" (UniqueName: \"kubernetes.io/projected/d24123dc-801b-41ea-953f-1adc49100354-kube-api-access-ntb98\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:36.146419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.146412    2579 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d24123dc-801b-41ea-953f-1adc49100354-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:36.712376 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.712341    2579 generic.go:356] "Generic (PLEG): container finished" podID="d24123dc-801b-41ea-953f-1adc49100354" containerID="a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625" exitCode=0
Sep 09 20:19:36.712548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.712390    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"
Sep 09 20:19:36.712548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.712429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" event={"ID":"d24123dc-801b-41ea-953f-1adc49100354","Type":"ContainerDied","Data":"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"}
Sep 09 20:19:36.712548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.712481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt" event={"ID":"d24123dc-801b-41ea-953f-1adc49100354","Type":"ContainerDied","Data":"bb10055675520987ce30f1d12be23c1d0ab5842171e1a5cecb5cd511d9346d80"}
Sep 09 20:19:36.712548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.712501    2579 scope.go:117] "RemoveContainer" containerID="a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"
Sep 09 20:19:36.721976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.721956    2579 scope.go:117] "RemoveContainer" containerID="a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"
Sep 09 20:19:36.722325 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:19:36.722305    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625\": container with ID starting with a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625 not found: ID does not exist" containerID="a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"
Sep 09 20:19:36.722375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.722333    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625"} err="failed to get container status \"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625\": rpc error: code = NotFound desc = could not find container \"a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625\": container with ID starting with a0359cd7fb681cea28c6e1d68125ca989bf9bf17e92b161d5fc8eae11f038625 not found: ID does not exist"
Sep 09 20:19:36.732479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.732453    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:36.736491 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:36.736467    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-s46gt"]
Sep 09 20:19:37.611383 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:37.611342    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24123dc-801b-41ea-953f-1adc49100354" path="/var/lib/kubelet/pods/d24123dc-801b-41ea-953f-1adc49100354/volumes"
Sep 09 20:19:38.042642 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.042596    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-7frmd"]
Sep 09 20:19:38.042997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.042979    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24123dc-801b-41ea-953f-1adc49100354" containerName="seaweedfs-tls-custom"
Sep 09 20:19:38.043082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.043000    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24123dc-801b-41ea-953f-1adc49100354" containerName="seaweedfs-tls-custom"
Sep 09 20:19:38.043082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.043066    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24123dc-801b-41ea-953f-1adc49100354" containerName="seaweedfs-tls-custom"
Sep 09 20:19:38.047452 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.047431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.049818 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.049792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 20:19:38.049941 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.049851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:19:38.049941 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.049872    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rm66\""
Sep 09 20:19:38.052168 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.052146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7frmd"]
Sep 09 20:19:38.166468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.166427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98plm\" (UniqueName: \"kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.166661 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.166516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.267787 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.267742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.267985 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.267795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98plm\" (UniqueName: \"kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.268161 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.268140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.276018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.275987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98plm\" (UniqueName: \"kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm\") pod \"s3-tls-init-custom-7frmd\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") " pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.364774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.364680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:38.635194 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.635118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7frmd"]
Sep 09 20:19:38.638239 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:19:38.638210    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod457db727_0f95_4d45_afe3_88f90b91a87b.slice/crio-fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed WatchSource:0}: Error finding container fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed: Status 404 returned error can't find the container with id fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed
Sep 09 20:19:38.724106 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.724059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerStarted","Data":"9bfd7f2adebae14162bb060274030fd155ae5ecc7025a4d1acee3fa7ee375ebb"}
Sep 09 20:19:38.724267 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:38.724124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerStarted","Data":"fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed"}
Sep 09 20:19:40.733833 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:40.733795    2579 generic.go:356] "Generic (PLEG): container finished" podID="457db727-0f95-4d45-afe3-88f90b91a87b" containerID="9bfd7f2adebae14162bb060274030fd155ae5ecc7025a4d1acee3fa7ee375ebb" exitCode=0
Sep 09 20:19:40.734292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:40.733862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerDied","Data":"9bfd7f2adebae14162bb060274030fd155ae5ecc7025a4d1acee3fa7ee375ebb"}
Sep 09 20:19:41.739628 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:41.739594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerStarted","Data":"918511c9f79ad48eb301f8876ec12b8d94eb9d097900110389407f000bfb11fd"}
Sep 09 20:19:41.756440 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:41.756382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-7frmd" podStartSLOduration=1.942837836 podStartE2EDuration="3.756363425s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.813525589s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:41.754669581 +0000 UTC m=+514.590984044" watchObservedRunningTime="2026-09-09 20:19:41.756363425 +0000 UTC m=+514.592677888"
Sep 09 20:19:47.767827 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:47.767790    2579 generic.go:356] "Generic (PLEG): container finished" podID="457db727-0f95-4d45-afe3-88f90b91a87b" containerID="918511c9f79ad48eb301f8876ec12b8d94eb9d097900110389407f000bfb11fd" exitCode=0
Sep 09 20:19:47.768237 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:47.767864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerDied","Data":"918511c9f79ad48eb301f8876ec12b8d94eb9d097900110389407f000bfb11fd"}
Sep 09 20:19:48.913299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:48.913272    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:49.061059 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.060965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98plm\" (UniqueName: \"kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm\") pod \"457db727-0f95-4d45-afe3-88f90b91a87b\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") "
Sep 09 20:19:49.061059 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.061025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache\") pod \"457db727-0f95-4d45-afe3-88f90b91a87b\" (UID: \"457db727-0f95-4d45-afe3-88f90b91a87b\") "
Sep 09 20:19:49.063267 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.063233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm" (OuterVolumeSpecName: "kube-api-access-98plm") pod "457db727-0f95-4d45-afe3-88f90b91a87b" (UID: "457db727-0f95-4d45-afe3-88f90b91a87b"). InnerVolumeSpecName "kube-api-access-98plm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:49.087749 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.087703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache" (OuterVolumeSpecName: "model-cache") pod "457db727-0f95-4d45-afe3-88f90b91a87b" (UID: "457db727-0f95-4d45-afe3-88f90b91a87b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:49.162457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.162427    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457db727-0f95-4d45-afe3-88f90b91a87b-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.162457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.162457    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98plm\" (UniqueName: \"kubernetes.io/projected/457db727-0f95-4d45-afe3-88f90b91a87b-kube-api-access-98plm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.777368 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.777341    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7frmd"
Sep 09 20:19:49.777516 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.777337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7frmd" event={"ID":"457db727-0f95-4d45-afe3-88f90b91a87b","Type":"ContainerDied","Data":"fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed"}
Sep 09 20:19:49.777516 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:49.777455    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4927b6046f921fa8627944cf1f4d82c8815ffd0c16b10adc1bc232fe8ab6ed"
Sep 09 20:19:50.370364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370332    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"]
Sep 09 20:19:50.370759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370603    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457db727-0f95-4d45-afe3-88f90b91a87b" containerName="download-hf-model"
Sep 09 20:19:50.370759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370614    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="457db727-0f95-4d45-afe3-88f90b91a87b" containerName="download-hf-model"
Sep 09 20:19:50.370759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370633    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457db727-0f95-4d45-afe3-88f90b91a87b" containerName="s3-tls-init-custom"
Sep 09 20:19:50.370759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370639    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="457db727-0f95-4d45-afe3-88f90b91a87b" containerName="s3-tls-init-custom"
Sep 09 20:19:50.370759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.370685    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="457db727-0f95-4d45-afe3-88f90b91a87b" containerName="s3-tls-init-custom"
Sep 09 20:19:50.376264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.376244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.379393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.379362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 09 20:19:50.379531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.379373    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 09 20:19:50.380349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.380330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rm66\""
Sep 09 20:19:50.381436 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.381417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"]
Sep 09 20:19:50.476878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.476825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.477061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.476896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.477061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.476914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4pr\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-kube-api-access-km4pr\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.577864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.577832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.577979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.577884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.577979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.577902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4pr\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-kube-api-access-km4pr\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.578068 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:19:50.578016    2579 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 09 20:19:50.578068 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:19:50.578033    2579 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5: secret "seaweedfs-tls-serving" not found
Sep 09 20:19:50.578211 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:19:50.578134    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving podName:86e0e92a-3373-47e2-beb3-3aa3fb4830dc nodeName:}" failed. No retries permitted until 2026-09-09 20:19:51.078113479 +0000 UTC m=+523.914427935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-4rdx5" (UID: "86e0e92a-3373-47e2-beb3-3aa3fb4830dc") : secret "seaweedfs-tls-serving" not found
Sep 09 20:19:50.578360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.578323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:50.587946 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:50.587916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4pr\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-kube-api-access-km4pr\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:51.082514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:51.082471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:51.085062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:51.085030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/86e0e92a-3373-47e2-beb3-3aa3fb4830dc-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4rdx5\" (UID: \"86e0e92a-3373-47e2-beb3-3aa3fb4830dc\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:51.287131 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:51.287073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"
Sep 09 20:19:51.559774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:51.559642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5"]
Sep 09 20:19:51.562419 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:19:51.562393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e0e92a_3373_47e2_beb3_3aa3fb4830dc.slice/crio-03b584108b4ae911f3f209b95c95d71f2afa455caeea47d4bf09c031a2ae8729 WatchSource:0}: Error finding container 03b584108b4ae911f3f209b95c95d71f2afa455caeea47d4bf09c031a2ae8729: Status 404 returned error can't find the container with id 03b584108b4ae911f3f209b95c95d71f2afa455caeea47d4bf09c031a2ae8729
Sep 09 20:19:51.787759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:51.787714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5" event={"ID":"86e0e92a-3373-47e2-beb3-3aa3fb4830dc","Type":"ContainerStarted","Data":"03b584108b4ae911f3f209b95c95d71f2afa455caeea47d4bf09c031a2ae8729"}
Sep 09 20:19:52.793492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:52.793452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5" event={"ID":"86e0e92a-3373-47e2-beb3-3aa3fb4830dc","Type":"ContainerStarted","Data":"46d7c23b8e35aebf838edb7b4f478899f4d9922c18689989afdb2d0364d81cc2"}
Sep 09 20:19:52.807811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:52.807753    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4rdx5" podStartSLOduration=2.5562665989999998 podStartE2EDuration="2.80773049s" totalImagesPullingTime="251.463891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:52.807618676 +0000 UTC m=+525.643933144" watchObservedRunningTime="2026-09-09 20:19:52.80773049 +0000 UTC m=+525.644044953"
Sep 09 20:19:53.378677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.378644    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-t8th9"]
Sep 09 20:19:53.383194 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.383176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.385547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.385526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:19:53.388656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.388631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-t8th9"]
Sep 09 20:19:53.501979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.501940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.502188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.502011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2v5k\" (UniqueName: \"kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.602469 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.602427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.602639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.602533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2v5k\" (UniqueName: \"kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.602908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.602886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.610842 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.610815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2v5k\" (UniqueName: \"kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k\") pod \"s3-tls-init-serving-t8th9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") " pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.700836 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.700802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:19:53.969082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:53.969003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-t8th9"]
Sep 09 20:19:53.972049 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:19:53.972018    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5d10841_e2fa_4a6b_9ef3_94d951276da9.slice/crio-7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533 WatchSource:0}: Error finding container 7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533: Status 404 returned error can't find the container with id 7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533
Sep 09 20:19:54.802472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:54.802439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerStarted","Data":"62e48882dcb6ab76671b4803b53aebb6e0842cb8b3ce9b69fd8845e46245bbc8"}
Sep 09 20:19:54.802723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:54.802483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerStarted","Data":"7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533"}
Sep 09 20:19:56.812298 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:56.812261    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerID="62e48882dcb6ab76671b4803b53aebb6e0842cb8b3ce9b69fd8845e46245bbc8" exitCode=0
Sep 09 20:19:56.812729 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:56.812342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerDied","Data":"62e48882dcb6ab76671b4803b53aebb6e0842cb8b3ce9b69fd8845e46245bbc8"}
Sep 09 20:19:57.817907 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:57.817874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerStarted","Data":"0e3e69417801ca1b8d6ad8192f55dd04fa2e4ede3e4049ab76733e084ab65e07"}
Sep 09 20:19:57.834535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:19:57.834480    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-t8th9" podStartSLOduration=3.052651727 podStartE2EDuration="4.834462338s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.781810611s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:57.833226969 +0000 UTC m=+530.669541433" watchObservedRunningTime="2026-09-09 20:19:57.834462338 +0000 UTC m=+530.670776821"
Sep 09 20:20:04.852358 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:04.852273    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerID="0e3e69417801ca1b8d6ad8192f55dd04fa2e4ede3e4049ab76733e084ab65e07" exitCode=0
Sep 09 20:20:04.852358 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:04.852325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerDied","Data":"0e3e69417801ca1b8d6ad8192f55dd04fa2e4ede3e4049ab76733e084ab65e07"}
Sep 09 20:20:05.996053 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:05.996019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:20:06.104697 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.104651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache\") pod \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") "
Sep 09 20:20:06.104697 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.104707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2v5k\" (UniqueName: \"kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k\") pod \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\" (UID: \"d5d10841-e2fa-4a6b-9ef3-94d951276da9\") "
Sep 09 20:20:06.106981 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.106907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k" (OuterVolumeSpecName: "kube-api-access-g2v5k") pod "d5d10841-e2fa-4a6b-9ef3-94d951276da9" (UID: "d5d10841-e2fa-4a6b-9ef3-94d951276da9"). InnerVolumeSpecName "kube-api-access-g2v5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:06.127558 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.127520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5d10841-e2fa-4a6b-9ef3-94d951276da9" (UID: "d5d10841-e2fa-4a6b-9ef3-94d951276da9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:06.205534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.205498    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d10841-e2fa-4a6b-9ef3-94d951276da9-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:06.205534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.205526    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2v5k\" (UniqueName: \"kubernetes.io/projected/d5d10841-e2fa-4a6b-9ef3-94d951276da9-kube-api-access-g2v5k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:06.862023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.861989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-t8th9" event={"ID":"d5d10841-e2fa-4a6b-9ef3-94d951276da9","Type":"ContainerDied","Data":"7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533"}
Sep 09 20:20:06.862023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.862008    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-t8th9"
Sep 09 20:20:06.862023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:06.862029    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d37a1f77bcd0cf34cb0e47ae3ee3786dd3ddf7ad38fc61777e43a1eabbcf533"
Sep 09 20:20:18.703976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.703941    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:20:18.704384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.704276    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerName="download-hf-model"
Sep 09 20:20:18.704384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.704291    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerName="download-hf-model"
Sep 09 20:20:18.704384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.704307    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerName="s3-tls-init-serving"
Sep 09 20:20:18.704384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.704312    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerName="s3-tls-init-serving"
Sep 09 20:20:18.704384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.704361    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d10841-e2fa-4a6b-9ef3-94d951276da9" containerName="s3-tls-init-serving"
Sep 09 20:20:18.710042 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.710020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.712862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.712841    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 09 20:20:18.712996 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.712876    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:20:18.713233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.713199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:20:18.713233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.713204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2gbsk\""
Sep 09 20:20:18.713410 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.713243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 09 20:20:18.713410 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.713271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:20:18.713410 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.713205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:20:18.717934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.717913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:20:18.807768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.807724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.807927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.807839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.807927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.807865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.807927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.807912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.808028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.807968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6s99\" (UniqueName: \"kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.874284 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.874242    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:18.878346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.878326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.880694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.880672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b30e22-kube-rbac-proxy-sar-config\""
Sep 09 20:20:18.880997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.880976    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-b30e22\""
Sep 09 20:20:18.881065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.880985    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-b30e22-dockercfg-pqdpv\""
Sep 09 20:20:18.881065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.881034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-b30e22-predictor-serving-cert\""
Sep 09 20:20:18.887223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.887202    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:18.908337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.908497 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.908497 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.908618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.908618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.908719 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.908719 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.908823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.908823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqd6\" (UniqueName: \"kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:18.908931 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s99\" (UniqueName: \"kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.908931 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:20:18.908863    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 09 20:20:18.909029 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.908931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.909029 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:20:18.908959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls podName:2e7ea048-76fb-4065-9f86-02188ba473ab nodeName:}" failed. No retries permitted until 2026-09-09 20:20:19.408935172 +0000 UTC m=+552.245249613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls") pod "isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 09 20:20:18.909138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.909082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.909241 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.909226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:18.917121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:18.917075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s99\" (UniqueName: \"kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:19.009700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.009595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.009700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.009670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.009923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.009722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.009923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.009781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.009923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.009854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqd6\" (UniqueName: \"kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.011101 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.011052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.011388 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.011361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.012009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.011982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.014928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.014905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.017967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.017945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqd6\" (UniqueName: \"kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6\") pod \"isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") " pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.189823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.189775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:19.413449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.413413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:19.415991 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.415968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58ff488b8-wv8cx\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:19.463465 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.463438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:19.465040 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:20:19.465016    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373a533f_c79b_4441_97a6_ee9d5698fbaa.slice/crio-a7363bf9a8cfacd3749fda7f0589fd395307f5cfddb7940c3d2e1e1f440879aa WatchSource:0}: Error finding container a7363bf9a8cfacd3749fda7f0589fd395307f5cfddb7940c3d2e1e1f440879aa: Status 404 returned error can't find the container with id a7363bf9a8cfacd3749fda7f0589fd395307f5cfddb7940c3d2e1e1f440879aa
Sep 09 20:20:19.620739 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.620706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:20:19.897400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.897370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:20:19.899078 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:20:19.899052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7ea048_76fb_4065_9f86_02188ba473ab.slice/crio-07623cd208135c06d6bfd53e6e5c89d1e83b69e8d358ba426bf6f3565eb0786b WatchSource:0}: Error finding container 07623cd208135c06d6bfd53e6e5c89d1e83b69e8d358ba426bf6f3565eb0786b: Status 404 returned error can't find the container with id 07623cd208135c06d6bfd53e6e5c89d1e83b69e8d358ba426bf6f3565eb0786b
Sep 09 20:20:19.910950 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.910922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerStarted","Data":"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"}
Sep 09 20:20:19.911082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.910999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerStarted","Data":"a7363bf9a8cfacd3749fda7f0589fd395307f5cfddb7940c3d2e1e1f440879aa"}
Sep 09 20:20:19.912278 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:19.912252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerStarted","Data":"07623cd208135c06d6bfd53e6e5c89d1e83b69e8d358ba426bf6f3565eb0786b"}
Sep 09 20:20:20.918381 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:20.918346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerStarted","Data":"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"}
Sep 09 20:20:22.927410 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:22.927380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/0.log"
Sep 09 20:20:22.927844 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:22.927419    2579 generic.go:356] "Generic (PLEG): container finished" podID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerID="979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea" exitCode=1
Sep 09 20:20:22.927844 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:22.927498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerDied","Data":"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"}
Sep 09 20:20:23.618997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:23.618962    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:23.933955 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:23.933930    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/0.log"
Sep 09 20:20:23.934337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:23.934008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerStarted","Data":"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"}
Sep 09 20:20:23.934337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:23.934126    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer" containerID="cri-o://264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133" gracePeriod=30
Sep 09 20:20:24.939911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:24.939872    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerID="fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94" exitCode=0
Sep 09 20:20:24.940375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:24.939949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerDied","Data":"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"}
Sep 09 20:20:28.823439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.823404    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:20:28.839863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.839826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:28.845491 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.845456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 20:20:28.845879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.845362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 09 20:20:28.848552 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.848478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:20:28.906291 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.905906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:28.906291 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.905959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:28.906291 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.906040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:28.906291 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.906134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:28.906291 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:28.906160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6d9c\" (UniqueName: \"kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007406 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6d9c\" (UniqueName: \"kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.007979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.007955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.008457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.008408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.008581 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.008454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.011552 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.011506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.016701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.016673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6d9c\" (UniqueName: \"kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hzfpx\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.153843 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.153805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:20:29.849852 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.849815    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/1.log"
Sep 09 20:20:29.850549 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.850368    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/0.log"
Sep 09 20:20:29.852788 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.852765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:29.918182 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location\") pod \"373a533f-c79b-4441-97a6-ee9d5698fbaa\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") "
Sep 09 20:20:29.918370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config\") pod \"373a533f-c79b-4441-97a6-ee9d5698fbaa\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") "
Sep 09 20:20:29.918370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls\") pod \"373a533f-c79b-4441-97a6-ee9d5698fbaa\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") "
Sep 09 20:20:29.918370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert\") pod \"373a533f-c79b-4441-97a6-ee9d5698fbaa\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") "
Sep 09 20:20:29.918370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jqd6\" (UniqueName: \"kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6\") pod \"373a533f-c79b-4441-97a6-ee9d5698fbaa\" (UID: \"373a533f-c79b-4441-97a6-ee9d5698fbaa\") "
Sep 09 20:20:29.918594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "373a533f-c79b-4441-97a6-ee9d5698fbaa" (UID: "373a533f-c79b-4441-97a6-ee9d5698fbaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:29.918594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918581    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/373a533f-c79b-4441-97a6-ee9d5698fbaa-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:29.918692 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "373a533f-c79b-4441-97a6-ee9d5698fbaa" (UID: "373a533f-c79b-4441-97a6-ee9d5698fbaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:29.918835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.918813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "373a533f-c79b-4441-97a6-ee9d5698fbaa" (UID: "373a533f-c79b-4441-97a6-ee9d5698fbaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:29.920987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.920960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "373a533f-c79b-4441-97a6-ee9d5698fbaa" (UID: "373a533f-c79b-4441-97a6-ee9d5698fbaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:29.920987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.920961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6" (OuterVolumeSpecName: "kube-api-access-9jqd6") pod "373a533f-c79b-4441-97a6-ee9d5698fbaa" (UID: "373a533f-c79b-4441-97a6-ee9d5698fbaa"). InnerVolumeSpecName "kube-api-access-9jqd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:29.978753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.978722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/1.log"
Sep 09 20:20:29.979538 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.979516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2_373a533f-c79b-4441-97a6-ee9d5698fbaa/storage-initializer/0.log"
Sep 09 20:20:29.979728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.979704    2579 generic.go:356] "Generic (PLEG): container finished" podID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerID="264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133" exitCode=1
Sep 09 20:20:29.980048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.980019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"
Sep 09 20:20:29.980048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.979887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerDied","Data":"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"}
Sep 09 20:20:29.980228 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.980201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2" event={"ID":"373a533f-c79b-4441-97a6-ee9d5698fbaa","Type":"ContainerDied","Data":"a7363bf9a8cfacd3749fda7f0589fd395307f5cfddb7940c3d2e1e1f440879aa"}
Sep 09 20:20:29.980621 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.980283    2579 scope.go:117] "RemoveContainer" containerID="264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"
Sep 09 20:20:29.989766 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.989745    2579 scope.go:117] "RemoveContainer" containerID="979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"
Sep 09 20:20:29.997464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.997448    2579 scope.go:117] "RemoveContainer" containerID="264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"
Sep 09 20:20:29.997734 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:20:29.997716    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133\": container with ID starting with 264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133 not found: ID does not exist" containerID="264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"
Sep 09 20:20:29.997777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.997744    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133"} err="failed to get container status \"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133\": rpc error: code = NotFound desc = could not find container \"264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133\": container with ID starting with 264ed84f6e8b00e516e015c83b7e80973fe4da95f96d1119a37328d47f0f8133 not found: ID does not exist"
Sep 09 20:20:29.997777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.997763    2579 scope.go:117] "RemoveContainer" containerID="979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"
Sep 09 20:20:29.998009 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:20:29.997990    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea\": container with ID starting with 979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea not found: ID does not exist" containerID="979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"
Sep 09 20:20:29.998044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:29.998016    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea"} err="failed to get container status \"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea\": rpc error: code = NotFound desc = could not find container \"979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea\": container with ID starting with 979dfe4d74caa4872ca48b36caceec98a541c2881d9485dbf3e7daf382d4a0ea not found: ID does not exist"
Sep 09 20:20:30.015396 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.015359    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:30.019312 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.019260    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:30.019447 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.019317    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/373a533f-c79b-4441-97a6-ee9d5698fbaa-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:30.019447 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.019334    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/373a533f-c79b-4441-97a6-ee9d5698fbaa-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:30.019447 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.019348    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jqd6\" (UniqueName: \"kubernetes.io/projected/373a533f-c79b-4441-97a6-ee9d5698fbaa-kube-api-access-9jqd6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:20:30.020849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.020821    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-b30e22-predictor-68b5d57f84-2d9w2"]
Sep 09 20:20:30.102648 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.102624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:20:30.105124 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:20:30.105079    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15f8a0c_96ad_4214_8cc0_0fb84cd688bf.slice/crio-a8ebd3d0614f3dfe2e8efa7a1ec72ce3e181ac59df806fe7889e533492717171 WatchSource:0}: Error finding container a8ebd3d0614f3dfe2e8efa7a1ec72ce3e181ac59df806fe7889e533492717171: Status 404 returned error can't find the container with id a8ebd3d0614f3dfe2e8efa7a1ec72ce3e181ac59df806fe7889e533492717171
Sep 09 20:20:30.986131 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.986076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerStarted","Data":"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"}
Sep 09 20:20:30.986511 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:30.986169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerStarted","Data":"a8ebd3d0614f3dfe2e8efa7a1ec72ce3e181ac59df806fe7889e533492717171"}
Sep 09 20:20:31.611897 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:31.611866    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" path="/var/lib/kubelet/pods/373a533f-c79b-4441-97a6-ee9d5698fbaa/volumes"
Sep 09 20:20:41.038892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:41.038799    2579 generic.go:356] "Generic (PLEG): container finished" podID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerID="cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4" exitCode=0
Sep 09 20:20:41.038892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:41.038872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerDied","Data":"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"}
Sep 09 20:20:41.041128 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:41.041106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerStarted","Data":"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"}
Sep 09 20:20:54.127342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:20:54.127296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerStarted","Data":"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"}
Sep 09 20:21:05.189394 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:05.189323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerStarted","Data":"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"}
Sep 09 20:21:05.189866 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:05.189547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerStarted","Data":"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"}
Sep 09 20:21:05.189941 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:05.189917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:21:05.210588 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:05.210518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podStartSLOduration=10.163727963 podStartE2EDuration="37.210497714s" totalImagesPullingTime="23.145141887s" totalInitContainerRuntime="3.901627864s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:05.209321722 +0000 UTC m=+598.045636186" watchObservedRunningTime="2026-09-09 20:21:05.210497714 +0000 UTC m=+598.046812178"
Sep 09 20:21:06.196559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.196515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerStarted","Data":"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"}
Sep 09 20:21:06.197017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.196957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:21:06.197017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.197009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:21:06.197173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.197033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:21:06.197173 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.197055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:21:06.197864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.197833    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:06.198364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.198333    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:06.199102 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.199061    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:06.201821 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.201802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:21:06.218034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:06.217987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podStartSLOduration=3.9857738510000003 podStartE2EDuration="48.217971419s" totalImagesPullingTime="40.303980584s" totalInitContainerRuntime="3.928216984s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:06.216228926 +0000 UTC m=+599.052543390" watchObservedRunningTime="2026-09-09 20:21:06.217971419 +0000 UTC m=+599.054285879"
Sep 09 20:21:07.201930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:07.201883    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:07.202441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:07.202018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:07.202563 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:07.202541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:08.205513 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:08.205471    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:08.205955 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:08.205785    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:12.206435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:12.206407    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:21:12.207040 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:12.207008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:18.205557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:18.205502    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:18.206051 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:18.206028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:22.207231 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:22.207181    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:28.205896 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:28.205841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:28.206412 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:28.206272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:32.207349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:32.207305    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:38.205849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:38.205794    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:38.206372 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:38.206344    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:42.207563 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:42.207518    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:48.205776 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:48.205728    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:48.208374 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:48.206217    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:21:52.207487 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:52.207430    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:21:58.205423 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:58.205368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:21:58.205854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:21:58.205829    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:02.207800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:02.207756    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:22:08.206329 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:08.206287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:08.206867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:08.206846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:12.207557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:12.207515    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:22:13.797570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.797521    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:22:13.798012 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.797762    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" containerID="cri-o://390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2" gracePeriod=30
Sep 09 20:22:13.798012 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.797817    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" containerID="cri-o://75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97" gracePeriod=30
Sep 09 20:22:13.798012 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.797830    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" containerID="cri-o://70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13" gracePeriod=30
Sep 09 20:22:13.816134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816103    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"]
Sep 09 20:22:13.816431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816418    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.816431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816432    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.816506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816448    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.816506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816453    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.816506 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816504    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.816600 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.816513    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="373a533f-c79b-4441-97a6-ee9d5698fbaa" containerName="storage-initializer"
Sep 09 20:22:13.819877 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.819855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:13.822155 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.822134    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 09 20:22:13.822257 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.822243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 09 20:22:13.830278 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.830253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"]
Sep 09 20:22:13.926392 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.926355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6m55\" (UniqueName: \"kubernetes.io/projected/6213c661-f8ab-48cd-982c-75851ecf0583-kube-api-access-x6m55\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:13.926535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.926393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:13.926535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.926446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:13.926535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.926512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6213c661-f8ab-48cd-982c-75851ecf0583-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:13.926535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:13.926529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6213c661-f8ab-48cd-982c-75851ecf0583-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027157 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6213c661-f8ab-48cd-982c-75851ecf0583-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027157 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6213c661-f8ab-48cd-982c-75851ecf0583-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6m55\" (UniqueName: \"kubernetes.io/projected/6213c661-f8ab-48cd-982c-75851ecf0583-kube-api-access-x6m55\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6213c661-f8ab-48cd-982c-75851ecf0583-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.027940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.028014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.027995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6213c661-f8ab-48cd-982c-75851ecf0583-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.029768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.029750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6213c661-f8ab-48cd-982c-75851ecf0583-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.034877 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.034841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6m55\" (UniqueName: \"kubernetes.io/projected/6213c661-f8ab-48cd-982c-75851ecf0583-kube-api-access-x6m55\") pod \"isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5\" (UID: \"6213c661-f8ab-48cd-982c-75851ecf0583\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.129762 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.129671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:14.403040 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.403014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"]
Sep 09 20:22:14.404496 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:22:14.404468    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6213c661_f8ab_48cd_982c_75851ecf0583.slice/crio-7dbe36503c80894990771ceff87948658963fba3b9e245a43e3322c3e2689c98 WatchSource:0}: Error finding container 7dbe36503c80894990771ceff87948658963fba3b9e245a43e3322c3e2689c98: Status 404 returned error can't find the container with id 7dbe36503c80894990771ceff87948658963fba3b9e245a43e3322c3e2689c98
Sep 09 20:22:14.477043 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.477007    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerID="70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13" exitCode=2
Sep 09 20:22:14.477238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.477081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerDied","Data":"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"}
Sep 09 20:22:14.478863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.478832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerStarted","Data":"827da91c60a96f1b72f4f377c2aab571d9c45a6a57d523f31e4822d17c78a197"}
Sep 09 20:22:14.478975 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:14.478905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerStarted","Data":"7dbe36503c80894990771ceff87948658963fba3b9e245a43e3322c3e2689c98"}
Sep 09 20:22:16.197413 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:16.197367    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:18.205543 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:18.205497    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:22:18.208205 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:18.208176    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:19.501912 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:19.501882    2579 generic.go:356] "Generic (PLEG): container finished" podID="6213c661-f8ab-48cd-982c-75851ecf0583" containerID="827da91c60a96f1b72f4f377c2aab571d9c45a6a57d523f31e4822d17c78a197" exitCode=0
Sep 09 20:22:19.502304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:19.501952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerDied","Data":"827da91c60a96f1b72f4f377c2aab571d9c45a6a57d523f31e4822d17c78a197"}
Sep 09 20:22:20.508423 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.508391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerStarted","Data":"9a907e7e295c2098dd16de8d860b8d08666aabfed51eb9fe227238fed50f9107"}
Sep 09 20:22:20.508847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.508463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerStarted","Data":"5fa96f2f4fd66f02fa6a775939bbab826348b097fb6227af90b9927652402159"}
Sep 09 20:22:20.508847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.508487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" event={"ID":"6213c661-f8ab-48cd-982c-75851ecf0583","Type":"ContainerStarted","Data":"711a9733ec6de258d0e234d9222725f23b2921864869b22a1372f89f8910534e"}
Sep 09 20:22:20.508966 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.508923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:20.509018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.508985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:20.509018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.509013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:20.510501 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.510467    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:20.511148 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.511127    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:20.533381 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:20.533324    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podStartSLOduration=3.483222348 podStartE2EDuration="7.533309535s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.050087187s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:20.531620422 +0000 UTC m=+673.367934885" watchObservedRunningTime="2026-09-09 20:22:20.533309535 +0000 UTC m=+673.369623998"
Sep 09 20:22:21.197161 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:21.197115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:21.512743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:21.512643    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:21.513142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:21.513111    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:22.208001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:22.207971    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:22:22.519165 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:22.519135    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerID="390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2" exitCode=0
Sep 09 20:22:22.519468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:22.519215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerDied","Data":"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"}
Sep 09 20:22:26.197720 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:26.197679    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:26.198133 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:26.197808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:26.517044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:26.516961    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5"
Sep 09 20:22:26.517632 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:26.517594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:26.517979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:26.517951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:28.206127 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:28.206060    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:22:28.207712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:28.207683    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:28.893274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:28.893240    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:22:28.893545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:28.893498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" containerID="cri-o://2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b" gracePeriod=30
Sep 09 20:22:28.893669 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:28.893529    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kube-rbac-proxy" containerID="cri-o://53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff" gracePeriod=30
Sep 09 20:22:29.551467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:29.551436    2579 generic.go:356] "Generic (PLEG): container finished" podID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerID="53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff" exitCode=2
Sep 09 20:22:29.551834 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:29.551506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerDied","Data":"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"}
Sep 09 20:22:31.197200 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:31.197162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:32.202222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:32.202177    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 09 20:22:32.207622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:32.207579    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 09 20:22:33.943207 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:33.943180    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:22:34.065707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.065628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:22:34.066003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.065979    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kube-rbac-proxy"
Sep 09 20:22:34.066003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.065997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kube-rbac-proxy"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066016    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066021    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066031    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="storage-initializer"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066037    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="storage-initializer"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066100    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kube-rbac-proxy"
Sep 09 20:22:34.066174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.066113    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerName="kserve-container"
Sep 09 20:22:34.069195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.069178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.071689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.071665    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 09 20:22:34.071822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.071665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 20:22:34.078945 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.078918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:22:34.108567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.108534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls\") pod \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") "
Sep 09 20:22:34.108755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.108639    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config\") pod \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") "
Sep 09 20:22:34.108755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.108667    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6d9c\" (UniqueName: \"kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c\") pod \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") "
Sep 09 20:22:34.108755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.108700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location\") pod \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") "
Sep 09 20:22:34.108918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.108729    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert\") pod \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\" (UID: \"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf\") "
Sep 09 20:22:34.109112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.109055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" (UID: "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:34.109310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.109076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" (UID: "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:34.109310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.109169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" (UID: "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:34.110840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.110815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" (UID: "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:34.110943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.110918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c" (OuterVolumeSpecName: "kube-api-access-m6d9c") pod "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" (UID: "d15f8a0c-96ad-4214-8cc0-0fb84cd688bf"). InnerVolumeSpecName "kube-api-access-m6d9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:34.209830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.209785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.209830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.209829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.210045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.209858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.210045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.209940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrt8\" (UniqueName: \"kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.210045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.210221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210060    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:34.210221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210079    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:34.210221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210117    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6d9c\" (UniqueName: \"kubernetes.io/projected/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kube-api-access-m6d9c\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:34.210221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210127    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:34.210221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.210135    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:34.311457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.311634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.311634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.311634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.311634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrt8\" (UniqueName: \"kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.312000 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.311974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.312213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.312191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.312345 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.312323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.314215 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.314197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.320154 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.320076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrt8\" (UniqueName: \"kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8\") pod \"isvc-predictive-xgboost-predictor-695f978978-98q4p\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.380773 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.380732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:34.574974 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.574891    2579 generic.go:356] "Generic (PLEG): container finished" podID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" containerID="2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b" exitCode=0
Sep 09 20:22:34.574974 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.574962    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"
Sep 09 20:22:34.575198 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.574964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerDied","Data":"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"}
Sep 09 20:22:34.575198 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.575104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx" event={"ID":"d15f8a0c-96ad-4214-8cc0-0fb84cd688bf","Type":"ContainerDied","Data":"a8ebd3d0614f3dfe2e8efa7a1ec72ce3e181ac59df806fe7889e533492717171"}
Sep 09 20:22:34.575198 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.575136    2579 scope.go:117] "RemoveContainer" containerID="53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"
Sep 09 20:22:34.584190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.584172    2579 scope.go:117] "RemoveContainer" containerID="2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"
Sep 09 20:22:34.592453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.592435    2579 scope.go:117] "RemoveContainer" containerID="cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"
Sep 09 20:22:34.598499 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.598479    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:22:34.601914 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.601893    2579 scope.go:117] "RemoveContainer" containerID="53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"
Sep 09 20:22:34.602316 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:34.602299    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff\": container with ID starting with 53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff not found: ID does not exist" containerID="53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"
Sep 09 20:22:34.602385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602326    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff"} err="failed to get container status \"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff\": rpc error: code = NotFound desc = could not find container \"53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff\": container with ID starting with 53b382aa6062cdd0c1c31de807ffa13b2578e4285b0acf2890a91c0998b2c1ff not found: ID does not exist"
Sep 09 20:22:34.602385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602346    2579 scope.go:117] "RemoveContainer" containerID="2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"
Sep 09 20:22:34.602524 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602505    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hzfpx"]
Sep 09 20:22:34.602649 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:34.602625    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b\": container with ID starting with 2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b not found: ID does not exist" containerID="2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"
Sep 09 20:22:34.602691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602660    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b"} err="failed to get container status \"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b\": rpc error: code = NotFound desc = could not find container \"2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b\": container with ID starting with 2da8a41de45a4248465ae9cae6968b5d4a486a11a5b4ad329e013d10a73c2c7b not found: ID does not exist"
Sep 09 20:22:34.602691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602677    2579 scope.go:117] "RemoveContainer" containerID="cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"
Sep 09 20:22:34.602949 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:34.602931    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4\": container with ID starting with cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4 not found: ID does not exist" containerID="cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"
Sep 09 20:22:34.603011 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.602954    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4"} err="failed to get container status \"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4\": rpc error: code = NotFound desc = could not find container \"cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4\": container with ID starting with cbc6c1ed135ba45dd8f750ca748d1de3fe3367e753c959b4cb514e96df527ea4 not found: ID does not exist"
Sep 09 20:22:34.654476 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:34.654450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:22:34.655966 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:22:34.655943    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a9cbec_16eb_4391_851b_f2ff2dd4d8b8.slice/crio-16d60381ca07108b111851fad92b7ad2e4671b335548eb0afb026cb8c3377cf9 WatchSource:0}: Error finding container 16d60381ca07108b111851fad92b7ad2e4671b335548eb0afb026cb8c3377cf9: Status 404 returned error can't find the container with id 16d60381ca07108b111851fad92b7ad2e4671b335548eb0afb026cb8c3377cf9
Sep 09 20:22:35.583118 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:35.583060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerStarted","Data":"5915eecdd7438550dbd4ca01ef0a21c6c82ba545678c553477d3b8b4583e7044"}
Sep 09 20:22:35.583599 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:35.583144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerStarted","Data":"16d60381ca07108b111851fad92b7ad2e4671b335548eb0afb026cb8c3377cf9"}
Sep 09 20:22:35.610813 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:35.610780    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15f8a0c-96ad-4214-8cc0-0fb84cd688bf" path="/var/lib/kubelet/pods/d15f8a0c-96ad-4214-8cc0-0fb84cd688bf/volumes"
Sep 09 20:22:36.197603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:36.197554    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:36.518388 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:36.518291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:36.518942 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:36.518913    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:38.206128 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.206059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:22:38.206548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.206227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:38.207794 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.207767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:38.207914 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.207875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:38.597389 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.597358    2579 generic.go:356] "Generic (PLEG): container finished" podID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerID="5915eecdd7438550dbd4ca01ef0a21c6c82ba545678c553477d3b8b4583e7044" exitCode=0
Sep 09 20:22:38.597555 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:38.597408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerDied","Data":"5915eecdd7438550dbd4ca01ef0a21c6c82ba545678c553477d3b8b4583e7044"}
Sep 09 20:22:39.604252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.604216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerStarted","Data":"15bdeea01df3f3ddf91880855595a7616ab11a2480a31c4e85b5488ed3fcec9a"}
Sep 09 20:22:39.604666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.604282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerStarted","Data":"d81beaaec316a7920bae1e66d5f01f5511f4a3de8481d8e12d4fa4015e858518"}
Sep 09 20:22:39.604666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.604597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:39.604666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.604646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:39.606140 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.606104    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:22:39.621940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:39.621886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podStartSLOduration=1.824578395 podStartE2EDuration="5.621869016s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797290621s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:39.620897783 +0000 UTC m=+692.457212246" watchObservedRunningTime="2026-09-09 20:22:39.621869016 +0000 UTC m=+692.458183478"
Sep 09 20:22:40.609280 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:40.609244    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:22:41.197535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:41.197492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:22:43.947895 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.947867    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:43.995196 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config\") pod \"2e7ea048-76fb-4065-9f86-02188ba473ab\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") "
Sep 09 20:22:43.995196 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location\") pod \"2e7ea048-76fb-4065-9f86-02188ba473ab\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") "
Sep 09 20:22:43.995437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert\") pod \"2e7ea048-76fb-4065-9f86-02188ba473ab\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") "
Sep 09 20:22:43.995437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995273    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6s99\" (UniqueName: \"kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99\") pod \"2e7ea048-76fb-4065-9f86-02188ba473ab\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") "
Sep 09 20:22:43.995437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") pod \"2e7ea048-76fb-4065-9f86-02188ba473ab\" (UID: \"2e7ea048-76fb-4065-9f86-02188ba473ab\") "
Sep 09 20:22:43.995603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e7ea048-76fb-4065-9f86-02188ba473ab" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:43.995603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e7ea048-76fb-4065-9f86-02188ba473ab" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:43.995691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.995636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e7ea048-76fb-4065-9f86-02188ba473ab" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:43.997567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.997543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e7ea048-76fb-4065-9f86-02188ba473ab" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:43.997664 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:43.997574    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99" (OuterVolumeSpecName: "kube-api-access-r6s99") pod "2e7ea048-76fb-4065-9f86-02188ba473ab" (UID: "2e7ea048-76fb-4065-9f86-02188ba473ab"). InnerVolumeSpecName "kube-api-access-r6s99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:44.096569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.096480    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ea048-76fb-4065-9f86-02188ba473ab-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:44.096569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.096512    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:44.096569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.096523    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ea048-76fb-4065-9f86-02188ba473ab-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:44.096569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.096532    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ea048-76fb-4065-9f86-02188ba473ab-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:44.096569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.096540    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6s99\" (UniqueName: \"kubernetes.io/projected/2e7ea048-76fb-4065-9f86-02188ba473ab-kube-api-access-r6s99\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:22:44.626835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.626802    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerID="75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97" exitCode=0
Sep 09 20:22:44.627030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.626868    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"
Sep 09 20:22:44.627030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.626890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerDied","Data":"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"}
Sep 09 20:22:44.627030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.626959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx" event={"ID":"2e7ea048-76fb-4065-9f86-02188ba473ab","Type":"ContainerDied","Data":"07623cd208135c06d6bfd53e6e5c89d1e83b69e8d358ba426bf6f3565eb0786b"}
Sep 09 20:22:44.627030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.626992    2579 scope.go:117] "RemoveContainer" containerID="75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"
Sep 09 20:22:44.635742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.635723    2579 scope.go:117] "RemoveContainer" containerID="70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"
Sep 09 20:22:44.643455 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.643421    2579 scope.go:117] "RemoveContainer" containerID="390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"
Sep 09 20:22:44.648540 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.648511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:22:44.651702 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.651683    2579 scope.go:117] "RemoveContainer" containerID="fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"
Sep 09 20:22:44.654116 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.654078    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58ff488b8-wv8cx"]
Sep 09 20:22:44.660019 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660001    2579 scope.go:117] "RemoveContainer" containerID="75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"
Sep 09 20:22:44.660393 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:44.660377    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97\": container with ID starting with 75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97 not found: ID does not exist" containerID="75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"
Sep 09 20:22:44.660442 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660403    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97"} err="failed to get container status \"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97\": rpc error: code = NotFound desc = could not find container \"75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97\": container with ID starting with 75e918fafdacaca0b2f2286c4a57558820262ca45b85cc8d3f746df1912c8e97 not found: ID does not exist"
Sep 09 20:22:44.660442 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660422    2579 scope.go:117] "RemoveContainer" containerID="70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"
Sep 09 20:22:44.660685 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:44.660668    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13\": container with ID starting with 70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13 not found: ID does not exist" containerID="70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"
Sep 09 20:22:44.660751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660691    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13"} err="failed to get container status \"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13\": rpc error: code = NotFound desc = could not find container \"70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13\": container with ID starting with 70a74a04dbccd9a854ab901948cc0b9f0a0e923cd03081b8753190dce3bfdc13 not found: ID does not exist"
Sep 09 20:22:44.660751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660704    2579 scope.go:117] "RemoveContainer" containerID="390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"
Sep 09 20:22:44.660937 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:44.660921    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2\": container with ID starting with 390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2 not found: ID does not exist" containerID="390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"
Sep 09 20:22:44.660976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2"} err="failed to get container status \"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2\": rpc error: code = NotFound desc = could not find container \"390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2\": container with ID starting with 390d93ae165d192f422ed27a156591721340f13b50cd2c37db93429d3d2e85b2 not found: ID does not exist"
Sep 09 20:22:44.660976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.660951    2579 scope.go:117] "RemoveContainer" containerID="fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"
Sep 09 20:22:44.661198 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:22:44.661182    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94\": container with ID starting with fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94 not found: ID does not exist" containerID="fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"
Sep 09 20:22:44.661274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:44.661201    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94"} err="failed to get container status \"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94\": rpc error: code = NotFound desc = could not find container \"fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94\": container with ID starting with fb81c9d19bd3830abecf136d4f6b6ead96cc84dd21fec2c595d490a424b1ed94 not found: ID does not exist"
Sep 09 20:22:45.610705 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:45.610666    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" path="/var/lib/kubelet/pods/2e7ea048-76fb-4065-9f86-02188ba473ab/volumes"
Sep 09 20:22:45.614386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:45.614364    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:22:45.614891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:45.614859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:22:46.517562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:46.517519    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:46.517981 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:46.517959    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:55.614800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:55.614755    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:22:56.518308 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:56.518259    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:22:56.519071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:22:56.519039    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:05.615891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:05.615805    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:23:06.518010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:06.517959    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:06.518443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:06.518418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:15.615015 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:15.614976    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:23:16.517669 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:16.517621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:16.518121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:16.518072    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:25.615068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:25.615027    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:23:26.518189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:26.518142    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:26.518515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:26.518486    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:35.615456 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:35.615412    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:23:36.517827 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:36.517778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:36.518308 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:36.518281    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:45.607065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:45.607008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:45.607531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:45.607258    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:45.615139 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:45.615079    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:23:55.607517 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:55.607466    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:23:55.608034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:55.608003    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:23:55.615824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:23:55.615799    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:24:04.186960 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:04.186928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:24:04.187388 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:04.187270    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" containerID="cri-o://d81beaaec316a7920bae1e66d5f01f5511f4a3de8481d8e12d4fa4015e858518" gracePeriod=30
Sep 09 20:24:04.187388 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:04.187291    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kube-rbac-proxy" containerID="cri-o://15bdeea01df3f3ddf91880855595a7616ab11a2480a31c4e85b5488ed3fcec9a" gracePeriod=30
Sep 09 20:24:04.964024 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:04.963992    2579 generic.go:356] "Generic (PLEG): container finished" podID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerID="15bdeea01df3f3ddf91880855595a7616ab11a2480a31c4e85b5488ed3fcec9a" exitCode=2
Sep 09 20:24:04.964238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:04.964072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerDied","Data":"15bdeea01df3f3ddf91880855595a7616ab11a2480a31c4e85b5488ed3fcec9a"}
Sep 09 20:24:05.607170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:05.607133    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:05.607637 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:05.607613    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:05.609634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:05.609603    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 09 20:24:05.614873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:05.614851    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:24:08.985460 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:08.985419    2579 generic.go:356] "Generic (PLEG): container finished" podID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerID="d81beaaec316a7920bae1e66d5f01f5511f4a3de8481d8e12d4fa4015e858518" exitCode=0
Sep 09 20:24:08.985995 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:08.985491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerDied","Data":"d81beaaec316a7920bae1e66d5f01f5511f4a3de8481d8e12d4fa4015e858518"}
Sep 09 20:24:09.024839 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.024813    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:24:09.059079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls\") pod \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") "
Sep 09 20:24:09.059289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config\") pod \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") "
Sep 09 20:24:09.059289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location\") pod \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") "
Sep 09 20:24:09.059289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert\") pod \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") "
Sep 09 20:24:09.059453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrt8\" (UniqueName: \"kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8\") pod \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\" (UID: \"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8\") "
Sep 09 20:24:09.059576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" (UID: "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:09.059641 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" (UID: "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:09.059641 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.059617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" (UID: "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:09.061385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.061359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" (UID: "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:09.061801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.061775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8" (OuterVolumeSpecName: "kube-api-access-5mrt8") pod "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" (UID: "34a9cbec-16eb-4391-851b-f2ff2dd4d8b8"). InnerVolumeSpecName "kube-api-access-5mrt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:09.160817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.160780    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrt8\" (UniqueName: \"kubernetes.io/projected/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-api-access-5mrt8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:24:09.160817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.160817    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:24:09.161010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.160828    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:24:09.161010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.160836    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:24:09.161010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.160846    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:24:09.355736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.355702    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:24:09.356007 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.355995    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356009    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356020    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356032    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356055    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356062    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="storage-initializer"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356067    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="storage-initializer"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356082    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="storage-initializer"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="storage-initializer"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356114    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356134    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356179    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kserve-container"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356186    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kserve-container"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356194    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356200    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="kube-rbac-proxy"
Sep 09 20:24:09.356276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.356207    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ea048-76fb-4065-9f86-02188ba473ab" containerName="agent"
Sep 09 20:24:09.359564 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.359548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.364675 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.364650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 09 20:24:09.364807 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.364675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 09 20:24:09.369658 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.369631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:24:09.463857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.463751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcnd\" (UniqueName: \"kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.463857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.463805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.463857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.463858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.464125 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.463906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.464125 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.463931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564331 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcnd\" (UniqueName: \"kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.564884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.564857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.565216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.565193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.565305 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.565198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.567386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.567359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.573146 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.573125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcnd\" (UniqueName: \"kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.670938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.670890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:09.942603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.942580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:24:09.944738 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:24:09.944706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc695b4c_dbf9_4638_b201_80ae11241bfd.slice/crio-78d04306aa240f508b35dee5d78f0117aee07d7ff15dcf509e967670fc5b94a4 WatchSource:0}: Error finding container 78d04306aa240f508b35dee5d78f0117aee07d7ff15dcf509e967670fc5b94a4: Status 404 returned error can't find the container with id 78d04306aa240f508b35dee5d78f0117aee07d7ff15dcf509e967670fc5b94a4
Sep 09 20:24:09.994758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.994729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p" event={"ID":"34a9cbec-16eb-4391-851b-f2ff2dd4d8b8","Type":"ContainerDied","Data":"16d60381ca07108b111851fad92b7ad2e4671b335548eb0afb026cb8c3377cf9"}
Sep 09 20:24:09.994758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.994748    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"
Sep 09 20:24:09.995257 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.994794    2579 scope.go:117] "RemoveContainer" containerID="15bdeea01df3f3ddf91880855595a7616ab11a2480a31c4e85b5488ed3fcec9a"
Sep 09 20:24:09.996239 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:09.996209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerStarted","Data":"78d04306aa240f508b35dee5d78f0117aee07d7ff15dcf509e967670fc5b94a4"}
Sep 09 20:24:10.012793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:10.012767    2579 scope.go:117] "RemoveContainer" containerID="d81beaaec316a7920bae1e66d5f01f5511f4a3de8481d8e12d4fa4015e858518"
Sep 09 20:24:10.021538 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:10.021502    2579 scope.go:117] "RemoveContainer" containerID="5915eecdd7438550dbd4ca01ef0a21c6c82ba545678c553477d3b8b4583e7044"
Sep 09 20:24:10.023763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:10.023735    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:24:10.027647 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:10.027623    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-98q4p"]
Sep 09 20:24:11.002267 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:11.002234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerStarted","Data":"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"}
Sep 09 20:24:11.610582 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:11.610546    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a9cbec-16eb-4391-851b-f2ff2dd4d8b8" path="/var/lib/kubelet/pods/34a9cbec-16eb-4391-851b-f2ff2dd4d8b8/volumes"
Sep 09 20:24:14.017364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:14.017327    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerID="bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6" exitCode=0
Sep 09 20:24:14.017847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:14.017396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerDied","Data":"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"}
Sep 09 20:24:15.023570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.023530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerStarted","Data":"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"}
Sep 09 20:24:15.024047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.023598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerStarted","Data":"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"}
Sep 09 20:24:15.024047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.023884    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:15.024047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.024012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:15.025468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.025439    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:15.043320 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.043273    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podStartSLOduration=2.213582133 podStartE2EDuration="6.043259981s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829677848s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:15.041814993 +0000 UTC m=+787.878129455" watchObservedRunningTime="2026-09-09 20:24:15.043259981 +0000 UTC m=+787.879574444"
Sep 09 20:24:15.607427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.607385    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:15.607920 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:15.607894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:16.027495 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:16.027459    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:21.032764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:21.032731    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:24:21.033295 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:21.033266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:25.607360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:25.607309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:25.607864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:25.607831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:31.033419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:31.033336    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:35.607493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:35.607449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:35.608059 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:35.608028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:41.033857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:41.033812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:45.607275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:45.607224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:45.607747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:45.607720    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:49.607017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:49.606962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:49.607541 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:49.607274    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:24:51.033283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:51.033242    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:24:59.607289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:59.607238    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:24:59.607777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:24:59.607695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:01.033572 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:01.033523    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:09.606964 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:09.606910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:09.607505 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:09.607428    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:11.033258 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:11.033214    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:19.607114 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:19.607046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:19.607549 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:19.607488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:21.033784 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:21.033742    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:29.606914 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:29.606857    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:29.607472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:29.607370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:31.034840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:31.034811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:25:39.441546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:39.441505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:25:39.442365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:39.441809    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" containerID="cri-o://7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec" gracePeriod=30
Sep 09 20:25:39.442365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:39.441886    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kube-rbac-proxy" containerID="cri-o://ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b" gracePeriod=30
Sep 09 20:25:39.607451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:39.607403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:39.607956 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:39.607930    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:40.372189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:40.372151    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerID="ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b" exitCode=2
Sep 09 20:25:40.372339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:40.372225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerDied","Data":"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"}
Sep 09 20:25:41.028359 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:41.028314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 09 20:25:41.033755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:41.033698    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:25:44.630780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.630741    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:25:44.634340 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.634322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.636671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.636646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:25:44.636784 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.636744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 09 20:25:44.645036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.645014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:25:44.789467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.789425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbx9\" (UniqueName: \"kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.789674 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.789496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.789674 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.789573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.789674 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.789640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.789817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.789685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.890811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.890788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.890943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.890830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.890943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.890860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.890943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.890895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.891055 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.890963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbx9\" (UniqueName: \"kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.891485 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.891455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.891608 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.891578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.891684 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.891664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.893408 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.893386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.898563 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.898533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbx9\" (UniqueName: \"kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:44.944587 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:44.944554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:45.173421 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.173395    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:25:45.217061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.217035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:25:45.219207 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:25:45.219179    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126e3bb4_e63a_4f06_9ad2_86217d0f978a.slice/crio-77473435d71281c4db6c8c3fff2ef24860140b1bfd0c47fd935c3d132d725a63 WatchSource:0}: Error finding container 77473435d71281c4db6c8c3fff2ef24860140b1bfd0c47fd935c3d132d725a63: Status 404 returned error can't find the container with id 77473435d71281c4db6c8c3fff2ef24860140b1bfd0c47fd935c3d132d725a63
Sep 09 20:25:45.294153 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls\") pod \"dc695b4c-dbf9-4638-b201-80ae11241bfd\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") "
Sep 09 20:25:45.294339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config\") pod \"dc695b4c-dbf9-4638-b201-80ae11241bfd\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") "
Sep 09 20:25:45.294414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert\") pod \"dc695b4c-dbf9-4638-b201-80ae11241bfd\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") "
Sep 09 20:25:45.294414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location\") pod \"dc695b4c-dbf9-4638-b201-80ae11241bfd\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") "
Sep 09 20:25:45.294522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxcnd\" (UniqueName: \"kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd\") pod \"dc695b4c-dbf9-4638-b201-80ae11241bfd\" (UID: \"dc695b4c-dbf9-4638-b201-80ae11241bfd\") "
Sep 09 20:25:45.294522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dc695b4c-dbf9-4638-b201-80ae11241bfd" (UID: "dc695b4c-dbf9-4638-b201-80ae11241bfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:45.294757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294737    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.294893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc695b4c-dbf9-4638-b201-80ae11241bfd" (UID: "dc695b4c-dbf9-4638-b201-80ae11241bfd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:45.294893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.294784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc695b4c-dbf9-4638-b201-80ae11241bfd" (UID: "dc695b4c-dbf9-4638-b201-80ae11241bfd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:45.296452 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.296426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc695b4c-dbf9-4638-b201-80ae11241bfd" (UID: "dc695b4c-dbf9-4638-b201-80ae11241bfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:45.296766 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.296739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd" (OuterVolumeSpecName: "kube-api-access-xxcnd") pod "dc695b4c-dbf9-4638-b201-80ae11241bfd" (UID: "dc695b4c-dbf9-4638-b201-80ae11241bfd"). InnerVolumeSpecName "kube-api-access-xxcnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:45.395270 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.395236    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc695b4c-dbf9-4638-b201-80ae11241bfd-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.395270 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.395267    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc695b4c-dbf9-4638-b201-80ae11241bfd-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.395507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.395280    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc695b4c-dbf9-4638-b201-80ae11241bfd-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.395507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.395294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxcnd\" (UniqueName: \"kubernetes.io/projected/dc695b4c-dbf9-4638-b201-80ae11241bfd-kube-api-access-xxcnd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.396763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.396736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerStarted","Data":"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"}
Sep 09 20:25:45.396873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.396832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerStarted","Data":"77473435d71281c4db6c8c3fff2ef24860140b1bfd0c47fd935c3d132d725a63"}
Sep 09 20:25:45.398832 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.398806    2579 generic.go:356] "Generic (PLEG): container finished" podID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerID="7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec" exitCode=0
Sep 09 20:25:45.398936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.398883    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"
Sep 09 20:25:45.398936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.398883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerDied","Data":"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"}
Sep 09 20:25:45.399014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.398940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c" event={"ID":"dc695b4c-dbf9-4638-b201-80ae11241bfd","Type":"ContainerDied","Data":"78d04306aa240f508b35dee5d78f0117aee07d7ff15dcf509e967670fc5b94a4"}
Sep 09 20:25:45.399014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.398977    2579 scope.go:117] "RemoveContainer" containerID="ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"
Sep 09 20:25:45.408835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.408818    2579 scope.go:117] "RemoveContainer" containerID="7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"
Sep 09 20:25:45.418516 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.418498    2579 scope.go:117] "RemoveContainer" containerID="bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"
Sep 09 20:25:45.426860 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.426829    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:25:45.427642 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.427623    2579 scope.go:117] "RemoveContainer" containerID="ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"
Sep 09 20:25:45.428153 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:25:45.428117    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b\": container with ID starting with ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b not found: ID does not exist" containerID="ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"
Sep 09 20:25:45.428261 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.428153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b"} err="failed to get container status \"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b\": rpc error: code = NotFound desc = could not find container \"ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b\": container with ID starting with ca93627a50d0d63fce1443666d1d322f6b63bbb1a54ecd39b43ce1000bf03d0b not found: ID does not exist"
Sep 09 20:25:45.428261 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.428182    2579 scope.go:117] "RemoveContainer" containerID="7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"
Sep 09 20:25:45.428582 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:25:45.428551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec\": container with ID starting with 7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec not found: ID does not exist" containerID="7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"
Sep 09 20:25:45.428687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.428589    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec"} err="failed to get container status \"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec\": rpc error: code = NotFound desc = could not find container \"7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec\": container with ID starting with 7566b4a4f57fc12418c86632013a2b67dcca2898704bb6d745e8bce7b59e2eec not found: ID does not exist"
Sep 09 20:25:45.428687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.428611    2579 scope.go:117] "RemoveContainer" containerID="bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"
Sep 09 20:25:45.428936 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:25:45.428918    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6\": container with ID starting with bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6 not found: ID does not exist" containerID="bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"
Sep 09 20:25:45.429005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.428945    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6"} err="failed to get container status \"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6\": rpc error: code = NotFound desc = could not find container \"bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6\": container with ID starting with bc16dab96a364f49bdb4a902223e3879481a87f942af35af6eaaada088bcb2d6 not found: ID does not exist"
Sep 09 20:25:45.430185 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.430164    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-dxl9c"]
Sep 09 20:25:45.610956 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:45.610924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" path="/var/lib/kubelet/pods/dc695b4c-dbf9-4638-b201-80ae11241bfd/volumes"
Sep 09 20:25:49.419487 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:49.419453    2579 generic.go:356] "Generic (PLEG): container finished" podID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerID="720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf" exitCode=0
Sep 09 20:25:49.419889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:49.419529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerDied","Data":"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"}
Sep 09 20:25:49.607373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:49.607324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:49.607850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:49.607820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:25:50.425813 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:50.425768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerStarted","Data":"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"}
Sep 09 20:25:50.426264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:50.425864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerStarted","Data":"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"}
Sep 09 20:25:50.426264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:50.426105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:50.426264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:50.426259    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:50.446509 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:50.446455    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podStartSLOduration=2.634293045 podStartE2EDuration="6.446438794s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812145749s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:50.445212994 +0000 UTC m=+883.281527459" watchObservedRunningTime="2026-09-09 20:25:50.446438794 +0000 UTC m=+883.282753258"
Sep 09 20:25:56.435889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:56.435857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:25:57.609602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:57.609556    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:25:57.610034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:25:57.609927    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:07.609744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:07.609621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:07.610550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:07.610526    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:17.609441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:17.609393    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:17.609993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:17.609912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:26.437327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:26.437279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:26:27.609772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:27.609724    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:27.610322 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:27.610288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:36.436851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:36.436783    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:26:37.609928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:37.609886    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:37.610480 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:37.610445    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:46.437149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:46.437078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:26:47.609912 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:47.609868    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:47.610478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:47.610454    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:26:56.436957 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:56.436912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:26:57.609406 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:57.609349    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:26:57.609910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:26:57.609756    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:01.606831 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:01.606743    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:01.607287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:01.607262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:04.606889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:04.606843    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:27:11.607138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:11.607068    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:11.607521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:11.607481    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:14.680292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:14.680257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:27:21.607014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:21.606964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:21.607546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:21.607460    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:24.718569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:24.718533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:27:24.718968 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:24.718807    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container" containerID="cri-o://fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1" gracePeriod=30
Sep 09 20:27:24.718968 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:24.718920    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kube-rbac-proxy" containerID="cri-o://4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197" gracePeriod=30
Sep 09 20:27:25.816366 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:25.816335    2579 generic.go:356] "Generic (PLEG): container finished" podID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerID="4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197" exitCode=2
Sep 09 20:27:25.816734 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:25.816410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerDied","Data":"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"}
Sep 09 20:27:26.430733 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:26.430688    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 20:27:29.664426 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.664401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:27:29.782627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls\") pod \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") "
Sep 09 20:27:29.782627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location\") pod \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") "
Sep 09 20:27:29.782861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert\") pod \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") "
Sep 09 20:27:29.782861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzbx9\" (UniqueName: \"kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9\") pod \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") "
Sep 09 20:27:29.782861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config\") pod \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\" (UID: \"126e3bb4-e63a-4f06-9ad2-86217d0f978a\") "
Sep 09 20:27:29.783031 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.782999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "126e3bb4-e63a-4f06-9ad2-86217d0f978a" (UID: "126e3bb4-e63a-4f06-9ad2-86217d0f978a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:29.783031 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.783019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "126e3bb4-e63a-4f06-9ad2-86217d0f978a" (UID: "126e3bb4-e63a-4f06-9ad2-86217d0f978a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:29.783160 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.783059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "126e3bb4-e63a-4f06-9ad2-86217d0f978a" (UID: "126e3bb4-e63a-4f06-9ad2-86217d0f978a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:29.784943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.784918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "126e3bb4-e63a-4f06-9ad2-86217d0f978a" (UID: "126e3bb4-e63a-4f06-9ad2-86217d0f978a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:29.785035 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.785000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9" (OuterVolumeSpecName: "kube-api-access-mzbx9") pod "126e3bb4-e63a-4f06-9ad2-86217d0f978a" (UID: "126e3bb4-e63a-4f06-9ad2-86217d0f978a"). InnerVolumeSpecName "kube-api-access-mzbx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:29.834791 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.834758    2579 generic.go:356] "Generic (PLEG): container finished" podID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerID="fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1" exitCode=0
Sep 09 20:27:29.834973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.834817    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"
Sep 09 20:27:29.834973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.834844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerDied","Data":"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"}
Sep 09 20:27:29.834973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.834921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9" event={"ID":"126e3bb4-e63a-4f06-9ad2-86217d0f978a","Type":"ContainerDied","Data":"77473435d71281c4db6c8c3fff2ef24860140b1bfd0c47fd935c3d132d725a63"}
Sep 09 20:27:29.834973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.834958    2579 scope.go:117] "RemoveContainer" containerID="4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"
Sep 09 20:27:29.843244 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.843224    2579 scope.go:117] "RemoveContainer" containerID="fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"
Sep 09 20:27:29.851671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.851652    2579 scope.go:117] "RemoveContainer" containerID="720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"
Sep 09 20:27:29.857290 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.857263    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:27:29.859943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.859927    2579 scope.go:117] "RemoveContainer" containerID="4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"
Sep 09 20:27:29.860324 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:27:29.860305    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197\": container with ID starting with 4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197 not found: ID does not exist" containerID="4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"
Sep 09 20:27:29.860414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.860338    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197"} err="failed to get container status \"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197\": rpc error: code = NotFound desc = could not find container \"4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197\": container with ID starting with 4dda9cfbd573b85c09522a6646c943d7c70321d943c3865d0f87969b2beea197 not found: ID does not exist"
Sep 09 20:27:29.860414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.860362    2579 scope.go:117] "RemoveContainer" containerID="fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"
Sep 09 20:27:29.860663 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:27:29.860646    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1\": container with ID starting with fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1 not found: ID does not exist" containerID="fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"
Sep 09 20:27:29.860707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.860669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1"} err="failed to get container status \"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1\": rpc error: code = NotFound desc = could not find container \"fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1\": container with ID starting with fe72ba6cdc94f6cf92314a460d5cb04513887882840b7f19a38f40cb4842aaf1 not found: ID does not exist"
Sep 09 20:27:29.860707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.860684    2579 scope.go:117] "RemoveContainer" containerID="720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"
Sep 09 20:27:29.860975 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:27:29.860953    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf\": container with ID starting with 720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf not found: ID does not exist" containerID="720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"
Sep 09 20:27:29.861139 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.860980    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf"} err="failed to get container status \"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf\": rpc error: code = NotFound desc = could not find container \"720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf\": container with ID starting with 720fb4a4a0eb6af6eaae94fa693e4783eb5ade4985754b0674aae35cd0cf3eaf not found: ID does not exist"
Sep 09 20:27:29.863225 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.863204    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-n9lg9"]
Sep 09 20:27:29.882197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882174    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:27:29.882473 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882461    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882486    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="storage-initializer"
Sep 09 20:27:29.882515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="storage-initializer"
Sep 09 20:27:29.882515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882499    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container"
Sep 09 20:27:29.882515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882505    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882523    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882532    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882537    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882546    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="storage-initializer"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882552    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="storage-initializer"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882594    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882601    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kserve-container"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882606    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" containerName="kube-rbac-proxy"
Sep 09 20:27:29.882663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.882614    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc695b4c-dbf9-4638-b201-80ae11241bfd" containerName="kserve-container"
Sep 09 20:27:29.884143 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.884124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbx9\" (UniqueName: \"kubernetes.io/projected/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-api-access-mzbx9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:27:29.884190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.884146    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:27:29.884190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.884156    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126e3bb4-e63a-4f06-9ad2-86217d0f978a-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:27:29.884190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.884165    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126e3bb4-e63a-4f06-9ad2-86217d0f978a-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:27:29.884190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.884173    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/126e3bb4-e63a-4f06-9ad2-86217d0f978a-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:27:29.887064 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.887048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:29.889461 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.889437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 09 20:27:29.889538 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.889467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:27:29.894349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.894218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:27:29.985447 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.985391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7vl\" (UniqueName: \"kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:29.985447 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.985445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:29.985692 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.985525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:29.985692 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.985564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:29.985692 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:29.985596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7vl\" (UniqueName: \"kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.086971 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.086943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.087360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.087339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.087418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.087343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.089230 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.089211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.094603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.094571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7vl\" (UniqueName: \"kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.198300 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.198247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:30.470293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.470260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:27:30.473113 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:27:30.473069    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8e757c_4374_4b40_b2b4_91b9953c2166.slice/crio-493fa73e174df9cfd1de0372316cc6c976b7a427b09f1ec1031cbe462654de3a WatchSource:0}: Error finding container 493fa73e174df9cfd1de0372316cc6c976b7a427b09f1ec1031cbe462654de3a: Status 404 returned error can't find the container with id 493fa73e174df9cfd1de0372316cc6c976b7a427b09f1ec1031cbe462654de3a
Sep 09 20:27:30.841427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.841340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerStarted","Data":"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"}
Sep 09 20:27:30.841427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:30.841403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerStarted","Data":"493fa73e174df9cfd1de0372316cc6c976b7a427b09f1ec1031cbe462654de3a"}
Sep 09 20:27:31.609264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:31.609208    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:31.612528 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:31.612488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:31.615424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:31.615398    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="126e3bb4-e63a-4f06-9ad2-86217d0f978a" path="/var/lib/kubelet/pods/126e3bb4-e63a-4f06-9ad2-86217d0f978a/volumes"
Sep 09 20:27:34.859246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:34.859212    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerID="c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953" exitCode=0
Sep 09 20:27:34.859667 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:34.859291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerDied","Data":"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"}
Sep 09 20:27:35.865679 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:35.865645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerStarted","Data":"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"}
Sep 09 20:27:35.866111 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:35.865717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerStarted","Data":"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"}
Sep 09 20:27:35.866111 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:35.865941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:35.866111 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:35.865995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:35.886050 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:35.885987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podStartSLOduration=3.27580081 podStartE2EDuration="6.885970319s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.610169509s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:35.883510121 +0000 UTC m=+988.719824613" watchObservedRunningTime="2026-09-09 20:27:35.885970319 +0000 UTC m=+988.722284782"
Sep 09 20:27:41.607330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:41.607280    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:41.607823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:41.607785    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:27:41.875057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:41.874969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:27:51.606724 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:51.606678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:27:51.607199 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:27:51.607114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:01.607214 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:01.607162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:01.607784 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:01.607684    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:11.606696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:11.606646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:11.607188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:11.607165    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:11.876201 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:11.876105    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:28:18.606495 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:18.606452    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:18.606931 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:18.606908    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:21.876334 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:21.876286    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:28:28.606561 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:28.606506    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:28.607019 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:28.606981    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:31.876608 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:31.876563    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:28:38.606468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:38.606411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:38.606970 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:38.606916    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:41.876479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:41.876442    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:28:48.607446 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:48.607393    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:48.607884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:48.607850    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:51.980302 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:51.980264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:28:58.606554 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:58.606457    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:58.607013 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:58.606989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:28:59.977579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:59.977543    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:28:59.977999 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:59.977811    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" containerID="cri-o://66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767" gracePeriod=30
Sep 09 20:28:59.977999 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:28:59.977871    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kube-rbac-proxy" containerID="cri-o://da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7" gracePeriod=30
Sep 09 20:29:00.215712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:00.215675    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerID="da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7" exitCode=2
Sep 09 20:29:00.215883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:00.215753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerDied","Data":"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"}
Sep 09 20:29:01.870682 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:01.870635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 09 20:29:01.875998 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:01.875963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:29:04.824430 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.824402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:29:04.887906 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.887880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls\") pod \"2b8e757c-4374-4b40-b2b4-91b9953c2166\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") "
Sep 09 20:29:04.888070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.887941    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location\") pod \"2b8e757c-4374-4b40-b2b4-91b9953c2166\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") "
Sep 09 20:29:04.888070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.887992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl7vl\" (UniqueName: \"kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl\") pod \"2b8e757c-4374-4b40-b2b4-91b9953c2166\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") "
Sep 09 20:29:04.888070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.888036    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config\") pod \"2b8e757c-4374-4b40-b2b4-91b9953c2166\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") "
Sep 09 20:29:04.888070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.888058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert\") pod \"2b8e757c-4374-4b40-b2b4-91b9953c2166\" (UID: \"2b8e757c-4374-4b40-b2b4-91b9953c2166\") "
Sep 09 20:29:04.888356 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.888268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b8e757c-4374-4b40-b2b4-91b9953c2166" (UID: "2b8e757c-4374-4b40-b2b4-91b9953c2166"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:04.888489 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.888430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b8e757c-4374-4b40-b2b4-91b9953c2166" (UID: "2b8e757c-4374-4b40-b2b4-91b9953c2166"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:04.888542 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.888480    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b8e757c-4374-4b40-b2b4-91b9953c2166" (UID: "2b8e757c-4374-4b40-b2b4-91b9953c2166"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:04.890037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.890017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b8e757c-4374-4b40-b2b4-91b9953c2166" (UID: "2b8e757c-4374-4b40-b2b4-91b9953c2166"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:04.890252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.890221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl" (OuterVolumeSpecName: "kube-api-access-cl7vl") pod "2b8e757c-4374-4b40-b2b4-91b9953c2166" (UID: "2b8e757c-4374-4b40-b2b4-91b9953c2166"). InnerVolumeSpecName "kube-api-access-cl7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:04.989222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.989144    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7vl\" (UniqueName: \"kubernetes.io/projected/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-api-access-cl7vl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:29:04.989222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.989169    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:29:04.989222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.989178    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b8e757c-4374-4b40-b2b4-91b9953c2166-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:29:04.989222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.989187    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b8e757c-4374-4b40-b2b4-91b9953c2166-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:29:04.989222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:04.989194    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b8e757c-4374-4b40-b2b4-91b9953c2166-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:29:05.132596 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.132553    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:29:05.132946 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.132929    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kube-rbac-proxy"
Sep 09 20:29:05.133046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.132949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kube-rbac-proxy"
Sep 09 20:29:05.133046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.132986    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container"
Sep 09 20:29:05.133046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.132995    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container"
Sep 09 20:29:05.133046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.133006    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="storage-initializer"
Sep 09 20:29:05.133046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.133015    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="storage-initializer"
Sep 09 20:29:05.133411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.133130    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kserve-container"
Sep 09 20:29:05.133411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.133144    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerName="kube-rbac-proxy"
Sep 09 20:29:05.136579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.136559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.139032 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.139014    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 09 20:29:05.139137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.139060    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:29:05.145745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.145724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:29:05.239686 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.239594    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b8e757c-4374-4b40-b2b4-91b9953c2166" containerID="66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767" exitCode=0
Sep 09 20:29:05.239686 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.239661    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"
Sep 09 20:29:05.239862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.239658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerDied","Data":"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"}
Sep 09 20:29:05.239862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.239795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54" event={"ID":"2b8e757c-4374-4b40-b2b4-91b9953c2166","Type":"ContainerDied","Data":"493fa73e174df9cfd1de0372316cc6c976b7a427b09f1ec1031cbe462654de3a"}
Sep 09 20:29:05.239862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.239834    2579 scope.go:117] "RemoveContainer" containerID="da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"
Sep 09 20:29:05.248580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.248562    2579 scope.go:117] "RemoveContainer" containerID="66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"
Sep 09 20:29:05.256120 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.256078    2579 scope.go:117] "RemoveContainer" containerID="c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"
Sep 09 20:29:05.260953 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.260928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:29:05.264379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.264357    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-p4w54"]
Sep 09 20:29:05.264659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.264634    2579 scope.go:117] "RemoveContainer" containerID="da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"
Sep 09 20:29:05.264999 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:29:05.264981    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7\": container with ID starting with da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7 not found: ID does not exist" containerID="da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"
Sep 09 20:29:05.265044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.265008    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7"} err="failed to get container status \"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7\": rpc error: code = NotFound desc = could not find container \"da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7\": container with ID starting with da43a9372417e58a76a2652a938264cf84d0a32b5115cfec60ed1c7238cf27a7 not found: ID does not exist"
Sep 09 20:29:05.265044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.265024    2579 scope.go:117] "RemoveContainer" containerID="66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"
Sep 09 20:29:05.265319 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:29:05.265303    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767\": container with ID starting with 66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767 not found: ID does not exist" containerID="66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"
Sep 09 20:29:05.265370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.265324    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767"} err="failed to get container status \"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767\": rpc error: code = NotFound desc = could not find container \"66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767\": container with ID starting with 66e9a94f690168924162665cd28e1375b64dadc584152eca7fb1d8cb33baa767 not found: ID does not exist"
Sep 09 20:29:05.265370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.265344    2579 scope.go:117] "RemoveContainer" containerID="c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"
Sep 09 20:29:05.265565 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:29:05.265550    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953\": container with ID starting with c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953 not found: ID does not exist" containerID="c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"
Sep 09 20:29:05.265610 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.265569    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953"} err="failed to get container status \"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953\": rpc error: code = NotFound desc = could not find container \"c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953\": container with ID starting with c7a4585b5b5a230b0883736631510189b6b3c46cc90e1b30492c3d92716ba953 not found: ID does not exist"
Sep 09 20:29:05.293012 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.292975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.293185 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.293023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.293239 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.293177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkt4c\" (UniqueName: \"kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.293239 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.293228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.293319 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.293296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.393970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkt4c\" (UniqueName: \"kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394304 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:29:05.394229    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 20:29:05.394304 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:29:05.394303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls podName:daa9482d-c6ed-4a07-b72e-fb74bf7e235d nodeName:}" failed. No retries permitted until 2026-09-09 20:29:05.894281167 +0000 UTC m=+1078.730595609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 20:29:05.394576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394832 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.394866 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.394810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.402563 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.402544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkt4c\" (UniqueName: \"kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.611962 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.611885    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b8e757c-4374-4b40-b2b4-91b9953c2166" path="/var/lib/kubelet/pods/2b8e757c-4374-4b40-b2b4-91b9953c2166/volumes"
Sep 09 20:29:05.899973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.899941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:05.902475 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:05.902447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:06.047676 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:06.047636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:06.323696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:06.322611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:29:07.250910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:07.250872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerStarted","Data":"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"}
Sep 09 20:29:07.251417 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:07.250932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerStarted","Data":"fd078d363ed5a50a0fd4820117ec20aec389251fd663c3d7f44ea7dcbcef6bb8"}
Sep 09 20:29:08.606768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:08.606723    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:08.607297 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:08.607272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:11.268654 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:11.268620    2579 generic.go:356] "Generic (PLEG): container finished" podID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerID="629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa" exitCode=0
Sep 09 20:29:11.269075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:11.268684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerDied","Data":"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"}
Sep 09 20:29:12.274781 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:12.274732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerStarted","Data":"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"}
Sep 09 20:29:12.275176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:12.274864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerStarted","Data":"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"}
Sep 09 20:29:12.275176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:12.275107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:12.294248 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:12.294201    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podStartSLOduration=3.155614322 podStartE2EDuration="7.294186063s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.138571741s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:12.291968326 +0000 UTC m=+1085.128282792" watchObservedRunningTime="2026-09-09 20:29:12.294186063 +0000 UTC m=+1085.130500526"
Sep 09 20:29:13.279824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:13.279789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:18.606491 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:18.606436    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:18.606953 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:18.606874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:19.289524 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:19.289494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:29:28.607174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:28.607126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:28.607634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:28.607532    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:37.609274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:37.609224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:37.609745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:37.609636    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:47.609164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:47.609078    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:47.609672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:47.609445    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:49.290624 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:49.290584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:29:57.609159 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:57.609111    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:57.609691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:57.609664    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:29:59.290528 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:29:59.290484    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:30:07.609635 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:07.609586    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:07.610845 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:07.610459    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:09.291029 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:09.290988    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:30:17.609777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:17.609735    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:17.610319 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:17.610291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:19.290892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:19.290852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:30:27.609170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:27.609118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:27.609639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:27.609531    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:29.380264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:29.380176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:30:35.236673 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:35.236638    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:30:35.237195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:35.236856    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" containerID="cri-o://e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d" gracePeriod=30
Sep 09 20:30:35.237195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:35.236900    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kube-rbac-proxy" containerID="cri-o://1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803" gracePeriod=30
Sep 09 20:30:35.619479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:35.619397    2579 generic.go:356] "Generic (PLEG): container finished" podID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerID="1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803" exitCode=2
Sep 09 20:30:35.619479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:35.619448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerDied","Data":"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"}
Sep 09 20:30:37.609800 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:37.609755    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:37.610387 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:37.610234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:39.284972 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:39.284924    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 09 20:30:39.290365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:39.290332    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:30:40.998650 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:40.998621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:30:41.112446 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert\") pod \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") "
Sep 09 20:30:41.112603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location\") pod \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") "
Sep 09 20:30:41.112603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config\") pod \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") "
Sep 09 20:30:41.112603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkt4c\" (UniqueName: \"kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c\") pod \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") "
Sep 09 20:30:41.112603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") pod \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\" (UID: \"daa9482d-c6ed-4a07-b72e-fb74bf7e235d\") "
Sep 09 20:30:41.112790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daa9482d-c6ed-4a07-b72e-fb74bf7e235d" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:41.112879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daa9482d-c6ed-4a07-b72e-fb74bf7e235d" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:41.112929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daa9482d-c6ed-4a07-b72e-fb74bf7e235d" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:41.112977 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.112945    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:30:41.114761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.114737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c" (OuterVolumeSpecName: "kube-api-access-kkt4c") pod "daa9482d-c6ed-4a07-b72e-fb74bf7e235d" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d"). InnerVolumeSpecName "kube-api-access-kkt4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:41.114825 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.114780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daa9482d-c6ed-4a07-b72e-fb74bf7e235d" (UID: "daa9482d-c6ed-4a07-b72e-fb74bf7e235d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:41.214163 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.214125    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:30:41.214163 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.214157    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:30:41.214163 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.214167    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkt4c\" (UniqueName: \"kubernetes.io/projected/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-kube-api-access-kkt4c\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:30:41.214397 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.214177    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa9482d-c6ed-4a07-b72e-fb74bf7e235d-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:30:41.644954 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.644916    2579 generic.go:356] "Generic (PLEG): container finished" podID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerID="e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d" exitCode=0
Sep 09 20:30:41.645138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.644990    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"
Sep 09 20:30:41.645138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.645004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerDied","Data":"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"}
Sep 09 20:30:41.645138 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.645121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v" event={"ID":"daa9482d-c6ed-4a07-b72e-fb74bf7e235d","Type":"ContainerDied","Data":"fd078d363ed5a50a0fd4820117ec20aec389251fd663c3d7f44ea7dcbcef6bb8"}
Sep 09 20:30:41.645252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.645155    2579 scope.go:117] "RemoveContainer" containerID="1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"
Sep 09 20:30:41.653465 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.653445    2579 scope.go:117] "RemoveContainer" containerID="e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"
Sep 09 20:30:41.661265 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.661240    2579 scope.go:117] "RemoveContainer" containerID="629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"
Sep 09 20:30:41.664036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.664010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:30:41.668456 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.668432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-crf2v"]
Sep 09 20:30:41.670327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.670311    2579 scope.go:117] "RemoveContainer" containerID="1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"
Sep 09 20:30:41.670657 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:30:41.670637    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803\": container with ID starting with 1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803 not found: ID does not exist" containerID="1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"
Sep 09 20:30:41.670712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.670669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803"} err="failed to get container status \"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803\": rpc error: code = NotFound desc = could not find container \"1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803\": container with ID starting with 1745f8aae65722b94cee16674649fb7b2cda44b20cd084ce8ed383eeac37a803 not found: ID does not exist"
Sep 09 20:30:41.670712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.670690    2579 scope.go:117] "RemoveContainer" containerID="e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"
Sep 09 20:30:41.670989 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:30:41.670970    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d\": container with ID starting with e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d not found: ID does not exist" containerID="e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"
Sep 09 20:30:41.671030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.670996    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d"} err="failed to get container status \"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d\": rpc error: code = NotFound desc = could not find container \"e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d\": container with ID starting with e5cfab955c0179deae62ede154eb38bbb49bf3c5d29bf55d6a76a5abb731ea0d not found: ID does not exist"
Sep 09 20:30:41.671030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.671013    2579 scope.go:117] "RemoveContainer" containerID="629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"
Sep 09 20:30:41.671340 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:30:41.671319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa\": container with ID starting with 629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa not found: ID does not exist" containerID="629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"
Sep 09 20:30:41.671404 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:41.671347    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa"} err="failed to get container status \"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa\": rpc error: code = NotFound desc = could not find container \"629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa\": container with ID starting with 629e0c284d4d84d0e5d0d53008ab8e9085c069a7af3890a93ccad0631297c5aa not found: ID does not exist"
Sep 09 20:30:43.611859 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:43.611825    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" path="/var/lib/kubelet/pods/daa9482d-c6ed-4a07-b72e-fb74bf7e235d/volumes"
Sep 09 20:30:46.606496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:46.606337    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:46.606918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:46.606778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:56.606467 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:56.606403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:30:56.606927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:56.606854    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:30:57.959945 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.959902    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-4fbf09e9"]
Sep 09 20:30:57.960450 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960429    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="storage-initializer"
Sep 09 20:30:57.960521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960449    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="storage-initializer"
Sep 09 20:30:57.960521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960467    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kube-rbac-proxy"
Sep 09 20:30:57.960521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960476    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kube-rbac-proxy"
Sep 09 20:30:57.960521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960503    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container"
Sep 09 20:30:57.960521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960513    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container"
Sep 09 20:30:57.960764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960592    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kserve-container"
Sep 09 20:30:57.960764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.960608    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa9482d-c6ed-4a07-b72e-fb74bf7e235d" containerName="kube-rbac-proxy"
Sep 09 20:30:57.965079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.965057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:30:57.969002 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:57.968972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-4fbf09e9"]
Sep 09 20:30:58.057449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.057411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbqs\" (UniqueName: \"kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs\") pod \"curl-metrics-4fbf09e9\" (UID: \"2dfd5a72-fc13-4a2d-8ad1-641539a401bd\") " pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:30:58.158282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.158241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbqs\" (UniqueName: \"kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs\") pod \"curl-metrics-4fbf09e9\" (UID: \"2dfd5a72-fc13-4a2d-8ad1-641539a401bd\") " pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:30:58.166747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.166716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbqs\" (UniqueName: \"kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs\") pod \"curl-metrics-4fbf09e9\" (UID: \"2dfd5a72-fc13-4a2d-8ad1-641539a401bd\") " pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:30:58.289279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.289187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:30:58.559735 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.559708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-4fbf09e9"]
Sep 09 20:30:58.562184 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:30:58.562153    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dfd5a72_fc13_4a2d_8ad1_641539a401bd.slice/crio-be92e3835e0b675ca62a051ccd9d1b3c802aa0cded5e1f9c684357098784d685 WatchSource:0}: Error finding container be92e3835e0b675ca62a051ccd9d1b3c802aa0cded5e1f9c684357098784d685: Status 404 returned error can't find the container with id be92e3835e0b675ca62a051ccd9d1b3c802aa0cded5e1f9c684357098784d685
Sep 09 20:30:58.564285 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.564269    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:30:58.714360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:58.714328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-4fbf09e9" event={"ID":"2dfd5a72-fc13-4a2d-8ad1-641539a401bd","Type":"ContainerStarted","Data":"be92e3835e0b675ca62a051ccd9d1b3c802aa0cded5e1f9c684357098784d685"}
Sep 09 20:30:59.719799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:59.719766    2579 generic.go:356] "Generic (PLEG): container finished" podID="2dfd5a72-fc13-4a2d-8ad1-641539a401bd" containerID="1e0e1e70b935e3b6340be02d535ace367fd67f7b4cfaccdeb4227186a8c4b259" exitCode=0
Sep 09 20:30:59.720239 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:30:59.719865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-4fbf09e9" event={"ID":"2dfd5a72-fc13-4a2d-8ad1-641539a401bd","Type":"ContainerDied","Data":"1e0e1e70b935e3b6340be02d535ace367fd67f7b4cfaccdeb4227186a8c4b259"}
Sep 09 20:31:00.855784 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.855755    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:31:00.889739 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.889706    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-4fbf09e9_2dfd5a72-fc13-4a2d-8ad1-641539a401bd/curl-metrics-4fbf09e9/0.log"
Sep 09 20:31:00.912034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.912001    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-4fbf09e9"]
Sep 09 20:31:00.913267 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.913246    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-4fbf09e9"]
Sep 09 20:31:00.984998 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.984909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbqs\" (UniqueName: \"kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs\") pod \"2dfd5a72-fc13-4a2d-8ad1-641539a401bd\" (UID: \"2dfd5a72-fc13-4a2d-8ad1-641539a401bd\") "
Sep 09 20:31:00.987221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:00.987194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs" (OuterVolumeSpecName: "kube-api-access-tbbqs") pod "2dfd5a72-fc13-4a2d-8ad1-641539a401bd" (UID: "2dfd5a72-fc13-4a2d-8ad1-641539a401bd"). InnerVolumeSpecName "kube-api-access-tbbqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:01.086528 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:01.086487    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbqs\" (UniqueName: \"kubernetes.io/projected/2dfd5a72-fc13-4a2d-8ad1-641539a401bd-kube-api-access-tbbqs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:31:01.610766 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:01.610729    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfd5a72-fc13-4a2d-8ad1-641539a401bd" path="/var/lib/kubelet/pods/2dfd5a72-fc13-4a2d-8ad1-641539a401bd/volumes"
Sep 09 20:31:01.729941 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:01.729909    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-4fbf09e9"
Sep 09 20:31:01.730157 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:01.729953    2579 scope.go:117] "RemoveContainer" containerID="1e0e1e70b935e3b6340be02d535ace367fd67f7b4cfaccdeb4227186a8c4b259"
Sep 09 20:31:06.343175 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.343139    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-5d225fe4"]
Sep 09 20:31:06.343622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.343438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfd5a72-fc13-4a2d-8ad1-641539a401bd" containerName="curl-metrics-4fbf09e9"
Sep 09 20:31:06.343622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.343451    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfd5a72-fc13-4a2d-8ad1-641539a401bd" containerName="curl-metrics-4fbf09e9"
Sep 09 20:31:06.343622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.343509    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfd5a72-fc13-4a2d-8ad1-641539a401bd" containerName="curl-metrics-4fbf09e9"
Sep 09 20:31:06.346386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.346371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:06.353736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.353712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-5d225fe4"]
Sep 09 20:31:06.426226 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.426186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h85j\" (UniqueName: \"kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j\") pod \"curl-metrics-5d225fe4\" (UID: \"4630bbcb-a498-45b3-9cd0-305295dafc45\") " pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:06.526687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.526657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h85j\" (UniqueName: \"kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j\") pod \"curl-metrics-5d225fe4\" (UID: \"4630bbcb-a498-45b3-9cd0-305295dafc45\") " pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:06.535332 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.535310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h85j\" (UniqueName: \"kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j\") pod \"curl-metrics-5d225fe4\" (UID: \"4630bbcb-a498-45b3-9cd0-305295dafc45\") " pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:06.606426 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.606329    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:06.606801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.606772    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:06.671075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.671035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:06.938900 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:06.938874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-5d225fe4"]
Sep 09 20:31:06.941073 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:31:06.941047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4630bbcb_a498_45b3_9cd0_305295dafc45.slice/crio-32058031660cb9dcd14ba2fba706848cf9fa33ba7bfddbe6d99fddb675da0daa WatchSource:0}: Error finding container 32058031660cb9dcd14ba2fba706848cf9fa33ba7bfddbe6d99fddb675da0daa: Status 404 returned error can't find the container with id 32058031660cb9dcd14ba2fba706848cf9fa33ba7bfddbe6d99fddb675da0daa
Sep 09 20:31:07.760862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:07.760826    2579 generic.go:356] "Generic (PLEG): container finished" podID="4630bbcb-a498-45b3-9cd0-305295dafc45" containerID="937584e728cbc6734225f6cb8d5b826cc02b400b14d51d4094b44aaa63a25c49" exitCode=0
Sep 09 20:31:07.761260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:07.760915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-5d225fe4" event={"ID":"4630bbcb-a498-45b3-9cd0-305295dafc45","Type":"ContainerDied","Data":"937584e728cbc6734225f6cb8d5b826cc02b400b14d51d4094b44aaa63a25c49"}
Sep 09 20:31:07.761260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:07.760960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-5d225fe4" event={"ID":"4630bbcb-a498-45b3-9cd0-305295dafc45","Type":"ContainerStarted","Data":"32058031660cb9dcd14ba2fba706848cf9fa33ba7bfddbe6d99fddb675da0daa"}
Sep 09 20:31:08.890171 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.890147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:08.909008 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.908982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-5d225fe4_4630bbcb-a498-45b3-9cd0-305295dafc45/curl-metrics-5d225fe4/0.log"
Sep 09 20:31:08.929199 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.929167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-5d225fe4"]
Sep 09 20:31:08.932782 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.932758    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-5d225fe4"]
Sep 09 20:31:08.945890 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.945863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h85j\" (UniqueName: \"kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j\") pod \"4630bbcb-a498-45b3-9cd0-305295dafc45\" (UID: \"4630bbcb-a498-45b3-9cd0-305295dafc45\") "
Sep 09 20:31:08.948172 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:08.948150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j" (OuterVolumeSpecName: "kube-api-access-9h85j") pod "4630bbcb-a498-45b3-9cd0-305295dafc45" (UID: "4630bbcb-a498-45b3-9cd0-305295dafc45"). InnerVolumeSpecName "kube-api-access-9h85j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:09.046904 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:09.046803    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h85j\" (UniqueName: \"kubernetes.io/projected/4630bbcb-a498-45b3-9cd0-305295dafc45-kube-api-access-9h85j\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:31:09.611733 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:09.611696    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4630bbcb-a498-45b3-9cd0-305295dafc45" path="/var/lib/kubelet/pods/4630bbcb-a498-45b3-9cd0-305295dafc45/volumes"
Sep 09 20:31:09.770550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:09.770519    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-5d225fe4"
Sep 09 20:31:09.770726 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:09.770533    2579 scope.go:117] "RemoveContainer" containerID="937584e728cbc6734225f6cb8d5b826cc02b400b14d51d4094b44aaa63a25c49"
Sep 09 20:31:14.282697 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.282658    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:31:14.283074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.282960    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4630bbcb-a498-45b3-9cd0-305295dafc45" containerName="curl-metrics-5d225fe4"
Sep 09 20:31:14.283074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.282971    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4630bbcb-a498-45b3-9cd0-305295dafc45" containerName="curl-metrics-5d225fe4"
Sep 09 20:31:14.283074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.283035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4630bbcb-a498-45b3-9cd0-305295dafc45" containerName="curl-metrics-5d225fe4"
Sep 09 20:31:14.287411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.287395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.289806 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.289785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 20:31:14.289806 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.289791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 09 20:31:14.295297 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.295272    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:31:14.390579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.390532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.390742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.390592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.390742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.390646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.390742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.390675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.390742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.390700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcfb\" (UniqueName: \"kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492171 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcfb\" (UniqueName: \"kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492661 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.492979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.492960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.493048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.493031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.495049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.495028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.501194 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.501171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcfb\" (UniqueName: \"kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb\") pod \"isvc-sklearn-predictor-bfb5f5bbd-2qwqm\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.599124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.599012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:14.873222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:14.872996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:31:14.875888 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:31:14.875855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d724d4_8f5a_4830_90bf_aca72fd96753.slice/crio-d114cca7cdb0345db5788c0c4b7e36a56531c3799cbe5035c45b092bb05e8966 WatchSource:0}: Error finding container d114cca7cdb0345db5788c0c4b7e36a56531c3799cbe5035c45b092bb05e8966: Status 404 returned error can't find the container with id d114cca7cdb0345db5788c0c4b7e36a56531c3799cbe5035c45b092bb05e8966
Sep 09 20:31:15.801489 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:15.801456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerStarted","Data":"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"}
Sep 09 20:31:15.801876 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:15.801514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerStarted","Data":"d114cca7cdb0345db5788c0c4b7e36a56531c3799cbe5035c45b092bb05e8966"}
Sep 09 20:31:16.607001 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:16.606955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:16.607475 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:16.607451    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:19.820049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:19.820011    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerID="5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be" exitCode=0
Sep 09 20:31:19.820458 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:19.820111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerDied","Data":"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"}
Sep 09 20:31:20.826412 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.826378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerStarted","Data":"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"}
Sep 09 20:31:20.826804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.826442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerStarted","Data":"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"}
Sep 09 20:31:20.826804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.826745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:20.826804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.826799    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:20.828533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.828499    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:20.846101 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:20.846042    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podStartSLOduration=2.824946326 podStartE2EDuration="6.846027333s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.021081007s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:20.843880693 +0000 UTC m=+1213.680195157" watchObservedRunningTime="2026-09-09 20:31:20.846027333 +0000 UTC m=+1213.682341795"
Sep 09 20:31:21.831044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:21.830999    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:26.607339 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:26.607286    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:26.607814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:26.607790    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:26.835542 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:26.835503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:31:26.836241 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:26.836209    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:36.606567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:36.606519    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:36.607023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:36.606775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:36.836935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:36.836897    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:46.606928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:46.606870    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:46.607411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:46.607307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:46.836225 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:46.836187    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:56.607044 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:56.606913    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:56.611309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:56.607773    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:31:56.836609 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:56.836571    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:31:57.608514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:57.608470    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:31:57.608968 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:31:57.608939    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:06.836368 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:06.836319    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:32:07.612377 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:07.612325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:07.614108 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:07.613113    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:16.837081 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:16.837039    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:32:17.608441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:17.608397    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:17.608948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:17.608918    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:26.836170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:26.836132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:32:27.608555 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:27.608500    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:27.609079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:27.609051    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:34.391164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.391121    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:32:34.396274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.396251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.398555 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.398529    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 09 20:32:34.398555 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.398551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 09 20:32:34.407080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.407050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:32:34.475409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.475358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6j9\" (UniqueName: \"kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.475409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.475410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.475694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.475515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.475694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.475579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.475694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.475659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.576922 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.576874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.576922 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.576919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6j9\" (UniqueName: \"kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577235 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577235 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577235 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577714 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.577757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.577725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.579724 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.579702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.585928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.585900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6j9\" (UniqueName: \"kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9\") pod \"isvc-logger-predictor-5b77679cf-kctj4\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.707309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.707200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:34.982416 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:34.982334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:32:34.985488 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:32:34.985458    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31109c8a_d680_40b1_a36e_6b95ec2afbe3.slice/crio-0f8a4365ebc849d130eee9c7841e437ad5dea554acbe101ef70d480836c7cb61 WatchSource:0}: Error finding container 0f8a4365ebc849d130eee9c7841e437ad5dea554acbe101ef70d480836c7cb61: Status 404 returned error can't find the container with id 0f8a4365ebc849d130eee9c7841e437ad5dea554acbe101ef70d480836c7cb61
Sep 09 20:32:35.128025 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:35.127981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerStarted","Data":"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"}
Sep 09 20:32:35.128244 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:35.128081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerStarted","Data":"0f8a4365ebc849d130eee9c7841e437ad5dea554acbe101ef70d480836c7cb61"}
Sep 09 20:32:36.837135 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:36.837099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:32:37.608782 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:37.608740    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:37.609352 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:37.609326    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:39.144804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:39.144771    2579 generic.go:356] "Generic (PLEG): container finished" podID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0" exitCode=0
Sep 09 20:32:39.145256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:39.144849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerDied","Data":"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"}
Sep 09 20:32:40.151434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.151400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerStarted","Data":"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"}
Sep 09 20:32:40.151854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.151470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerStarted","Data":"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"}
Sep 09 20:32:40.151854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.151497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerStarted","Data":"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"}
Sep 09 20:32:40.151854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.151793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:40.152022 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.151932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:40.153240 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.153214    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:32:40.174885 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:40.174828    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podStartSLOduration=2.184529988 podStartE2EDuration="6.174809946s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990279958s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:40.172857625 +0000 UTC m=+1293.009172080" watchObservedRunningTime="2026-09-09 20:32:40.174809946 +0000 UTC m=+1293.011124409"
Sep 09 20:32:41.156691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:41.156654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:41.157123 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:41.156837    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:32:41.157973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:41.157943    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:42.160619 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:42.160571    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:32:42.161035 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:42.160987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:44.359918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:44.359872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:32:44.360420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:44.360168    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" containerID="cri-o://a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4" gracePeriod=30
Sep 09 20:32:44.360420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:44.360217    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kube-rbac-proxy" containerID="cri-o://99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac" gracePeriod=30
Sep 09 20:32:45.175160 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:45.175123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerDied","Data":"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"}
Sep 09 20:32:45.175347 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:45.175123    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerID="99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac" exitCode=2
Sep 09 20:32:46.831783 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:46.831729    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 09 20:32:46.837137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:46.837076    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 09 20:32:47.165405 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:47.165378    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:32:47.166071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:47.166035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:32:47.166526 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:47.166501    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:47.608511 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:47.608396    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:47.608927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:47.608898    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:49.506638 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.506611    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:32:49.540519 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540438    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:32:49.540771 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540759    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kube-rbac-proxy"
Sep 09 20:32:49.540814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540774    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kube-rbac-proxy"
Sep 09 20:32:49.540814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540792    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container"
Sep 09 20:32:49.540814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container"
Sep 09 20:32:49.540814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540805    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="storage-initializer"
Sep 09 20:32:49.540814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540810    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="storage-initializer"
Sep 09 20:32:49.541076 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540861    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kube-rbac-proxy"
Sep 09 20:32:49.541076 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.540871    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerName="kserve-container"
Sep 09 20:32:49.543945 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.543928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.546543 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.546511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 09 20:32:49.546689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.546574    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:32:49.556313 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.556288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:32:49.597720 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.597679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location\") pod \"d7d724d4-8f5a-4830-90bf-aca72fd96753\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") "
Sep 09 20:32:49.597927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.597731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config\") pod \"d7d724d4-8f5a-4830-90bf-aca72fd96753\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") "
Sep 09 20:32:49.597927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.597880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kz5b\" (UniqueName: \"kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.597927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.597921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.598079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.597962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.598079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.598032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.598079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.598040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7d724d4-8f5a-4830-90bf-aca72fd96753" (UID: "d7d724d4-8f5a-4830-90bf-aca72fd96753"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:49.598247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.598144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.598247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.598182    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7d724d4-8f5a-4830-90bf-aca72fd96753-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.598247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.598186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7d724d4-8f5a-4830-90bf-aca72fd96753" (UID: "d7d724d4-8f5a-4830-90bf-aca72fd96753"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:49.698472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htcfb\" (UniqueName: \"kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb\") pod \"d7d724d4-8f5a-4830-90bf-aca72fd96753\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") "
Sep 09 20:32:49.698472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert\") pod \"d7d724d4-8f5a-4830-90bf-aca72fd96753\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") "
Sep 09 20:32:49.698717 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls\") pod \"d7d724d4-8f5a-4830-90bf-aca72fd96753\" (UID: \"d7d724d4-8f5a-4830-90bf-aca72fd96753\") "
Sep 09 20:32:49.698717 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.698717 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.698717 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.698926 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.698926 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kz5b\" (UniqueName: \"kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.698926 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698872    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.699147 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.698933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7d724d4-8f5a-4830-90bf-aca72fd96753" (UID: "d7d724d4-8f5a-4830-90bf-aca72fd96753"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:49.699147 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.699045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.699560 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.699535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.699678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.699590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.700930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.700904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb" (OuterVolumeSpecName: "kube-api-access-htcfb") pod "d7d724d4-8f5a-4830-90bf-aca72fd96753" (UID: "d7d724d4-8f5a-4830-90bf-aca72fd96753"). InnerVolumeSpecName "kube-api-access-htcfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:49.701035 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.700978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7d724d4-8f5a-4830-90bf-aca72fd96753" (UID: "d7d724d4-8f5a-4830-90bf-aca72fd96753"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:49.701724 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.701706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.706803 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.706778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kz5b\" (UniqueName: \"kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:49.800024 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.799933    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htcfb\" (UniqueName: \"kubernetes.io/projected/d7d724d4-8f5a-4830-90bf-aca72fd96753-kube-api-access-htcfb\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.800024 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.799969    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7d724d4-8f5a-4830-90bf-aca72fd96753-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.800024 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.799977    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7d724d4-8f5a-4830-90bf-aca72fd96753-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.853940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:49.853907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:32:50.131023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.130998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:32:50.132944 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:32:50.132900    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e62a33_91ed_451a_b7e1_7183cbd2e627.slice/crio-3be653b1ef5aae83a43aea45b8406db64ea8d7a5756f1ca2a1af84aba67f28e8 WatchSource:0}: Error finding container 3be653b1ef5aae83a43aea45b8406db64ea8d7a5756f1ca2a1af84aba67f28e8: Status 404 returned error can't find the container with id 3be653b1ef5aae83a43aea45b8406db64ea8d7a5756f1ca2a1af84aba67f28e8
Sep 09 20:32:50.199535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.199502    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7d724d4-8f5a-4830-90bf-aca72fd96753" containerID="a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4" exitCode=0
Sep 09 20:32:50.199723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.199552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerDied","Data":"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"}
Sep 09 20:32:50.199723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.199591    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"
Sep 09 20:32:50.199723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.199640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm" event={"ID":"d7d724d4-8f5a-4830-90bf-aca72fd96753","Type":"ContainerDied","Data":"d114cca7cdb0345db5788c0c4b7e36a56531c3799cbe5035c45b092bb05e8966"}
Sep 09 20:32:50.199723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.199667    2579 scope.go:117] "RemoveContainer" containerID="99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"
Sep 09 20:32:50.201470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.201400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerStarted","Data":"205c6e89558cfc3377500421db879e8f7fe6caf40b3c08d104f814e6c3a399f4"}
Sep 09 20:32:50.201470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.201462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerStarted","Data":"3be653b1ef5aae83a43aea45b8406db64ea8d7a5756f1ca2a1af84aba67f28e8"}
Sep 09 20:32:50.208178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.208160    2579 scope.go:117] "RemoveContainer" containerID="a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"
Sep 09 20:32:50.216368 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.216348    2579 scope.go:117] "RemoveContainer" containerID="5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"
Sep 09 20:32:50.225789 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.225762    2579 scope.go:117] "RemoveContainer" containerID="99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"
Sep 09 20:32:50.226163 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:32:50.226140    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac\": container with ID starting with 99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac not found: ID does not exist" containerID="99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"
Sep 09 20:32:50.226272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.226186    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac"} err="failed to get container status \"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac\": rpc error: code = NotFound desc = could not find container \"99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac\": container with ID starting with 99ca4a81bab37886369799afe7b4878126bfbc23ee2a8211888f6e28b1a109ac not found: ID does not exist"
Sep 09 20:32:50.226272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.226209    2579 scope.go:117] "RemoveContainer" containerID="a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"
Sep 09 20:32:50.226611 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:32:50.226583    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4\": container with ID starting with a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4 not found: ID does not exist" containerID="a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"
Sep 09 20:32:50.226668 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.226621    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4"} err="failed to get container status \"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4\": rpc error: code = NotFound desc = could not find container \"a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4\": container with ID starting with a5b711e265d6c575eeb64a4ae68b4b0908dd56f016fda3cd9b1a2e07f25e47c4 not found: ID does not exist"
Sep 09 20:32:50.226668 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.226645    2579 scope.go:117] "RemoveContainer" containerID="5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"
Sep 09 20:32:50.226952 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:32:50.226935    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be\": container with ID starting with 5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be not found: ID does not exist" containerID="5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"
Sep 09 20:32:50.227011 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.226958    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be"} err="failed to get container status \"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be\": rpc error: code = NotFound desc = could not find container \"5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be\": container with ID starting with 5683871e487b491186259c38584aeca2d346cdfece224ca24272259a4270c9be not found: ID does not exist"
Sep 09 20:32:50.233229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.233203    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:32:50.241846 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:50.241819    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bfb5f5bbd-2qwqm"]
Sep 09 20:32:51.610910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:51.610875    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d724d4-8f5a-4830-90bf-aca72fd96753" path="/var/lib/kubelet/pods/d7d724d4-8f5a-4830-90bf-aca72fd96753/volumes"
Sep 09 20:32:54.220372 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:54.220341    2579 generic.go:356] "Generic (PLEG): container finished" podID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerID="205c6e89558cfc3377500421db879e8f7fe6caf40b3c08d104f814e6c3a399f4" exitCode=0
Sep 09 20:32:54.220723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:54.220420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerDied","Data":"205c6e89558cfc3377500421db879e8f7fe6caf40b3c08d104f814e6c3a399f4"}
Sep 09 20:32:57.166274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:57.165955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:32:57.166928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:57.166903    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:32:57.608662 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:57.608476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:32:57.609349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:32:57.609152    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:07.166893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:07.166850    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:07.167430 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:07.167395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:07.608621 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:07.608528    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:07.609613 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:07.609412    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:11.606066 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:11.606018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:11.606647 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:11.606359    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:17.165966 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:17.165921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:17.166427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:17.166403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:21.606527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:21.606476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:21.606907 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:21.606490    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:27.165937 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:27.165892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:27.166464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:27.166380    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:31.606224 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:31.606126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:31.607011 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:31.606978    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:37.166690 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:37.166620    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:37.167237 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:37.167211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:41.606466 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:41.606422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:41.607006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:41.606979    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:47.166562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:47.166492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:47.167292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:47.167014    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:51.606579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:51.606529    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:33:51.607367 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:51.607324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:33:56.606888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:56.606849    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:33:56.607383 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:33:56.607229    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:01.606534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:01.606473    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:01.607107 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:01.607059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:06.606738 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:06.606692    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:34:06.607283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:06.607250    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:11.606837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:11.606765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:11.607322 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:11.607194    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:15.606598 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:15.606548    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:15.607080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:15.607025    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:16.607851 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:16.607814    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:34:16.608336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:16.607920    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:34:25.606840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:25.606781    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:25.607471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:25.607394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:29.625905 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.625822    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:34:29.626386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.626173    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" containerID="cri-o://aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d" gracePeriod=30
Sep 09 20:34:29.626386 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.626309    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" containerID="cri-o://2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550" gracePeriod=30
Sep 09 20:34:29.626583 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.626384    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" containerID="cri-o://b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65" gracePeriod=30
Sep 09 20:34:29.758416 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.758347    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:34:29.905328 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.905281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:34:29.905551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.905396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:29.908259 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.908233    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 09 20:34:29.908259 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:29.908247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 09 20:34:30.019936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.019888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.020151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.019958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.020151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.020014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.020250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.020142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.020250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.020205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7k67\" (UniqueName: \"kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121076 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7k67\" (UniqueName: \"kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.121821 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.121782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.122147 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.122074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.122268 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.122235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.124128 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.124103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.131436 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.131407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7k67\" (UniqueName: \"kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jb8tt\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:30.219000 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:30.218919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:34:32.161556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:32.161507    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:34:35.607406 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:35.607325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:35.607963 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:35.607846    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:36.607225 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:36.607172    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:34:36.607586 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:36.607535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:37.160936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:37.160889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:34:42.160798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:42.160748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:34:42.161405 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:42.160913    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:34:45.607370 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:45.607297    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:45.608120 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:45.607987    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:46.607724 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:46.607678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:34:46.608402 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:46.608358    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:47.161181 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:47.161121    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:34:52.161074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:52.161018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:34:55.607097 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:55.607040    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:34:55.607660 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:55.607621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:56.607557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:56.607513    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 09 20:34:56.608057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:56.607690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:34:56.608057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:56.608000    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:34:56.608211 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:56.608127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:34:57.160996 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:34:57.160959    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:35:02.161644 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:02.161591    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 09 20:35:03.212299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.212270    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:35:03.236958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.236923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls\") pod \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") "
Sep 09 20:35:03.237183 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.236981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location\") pod \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") "
Sep 09 20:35:03.237183 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config\") pod \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") "
Sep 09 20:35:03.237183 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert\") pod \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") "
Sep 09 20:35:03.237183 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz6j9\" (UniqueName: \"kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9\") pod \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\" (UID: \"31109c8a-d680-40b1-a36e-6b95ec2afbe3\") "
Sep 09 20:35:03.237419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31109c8a-d680-40b1-a36e-6b95ec2afbe3" (UID: "31109c8a-d680-40b1-a36e-6b95ec2afbe3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:03.237476 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31109c8a-d680-40b1-a36e-6b95ec2afbe3" (UID: "31109c8a-d680-40b1-a36e-6b95ec2afbe3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:03.237611 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.237583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31109c8a-d680-40b1-a36e-6b95ec2afbe3" (UID: "31109c8a-d680-40b1-a36e-6b95ec2afbe3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:03.239398 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.239372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31109c8a-d680-40b1-a36e-6b95ec2afbe3" (UID: "31109c8a-d680-40b1-a36e-6b95ec2afbe3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:03.239669 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.239638    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9" (OuterVolumeSpecName: "kube-api-access-jz6j9") pod "31109c8a-d680-40b1-a36e-6b95ec2afbe3" (UID: "31109c8a-d680-40b1-a36e-6b95ec2afbe3"). InnerVolumeSpecName "kube-api-access-jz6j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:03.338418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.338380    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.338418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.338409    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31109c8a-d680-40b1-a36e-6b95ec2afbe3-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.338418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.338418    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz6j9\" (UniqueName: \"kubernetes.io/projected/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kube-api-access-jz6j9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.338634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.338427    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31109c8a-d680-40b1-a36e-6b95ec2afbe3-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.338634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.338438    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31109c8a-d680-40b1-a36e-6b95ec2afbe3-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.357618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.357592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:35:03.359072 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:35:03.359046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab97367_a1c2_4195_a6f6_55d21692e01e.slice/crio-03fadc86df36da8f05bffbb231b30d787e2dcb4d5ddd356c4fcc9f38467b5248 WatchSource:0}: Error finding container 03fadc86df36da8f05bffbb231b30d787e2dcb4d5ddd356c4fcc9f38467b5248: Status 404 returned error can't find the container with id 03fadc86df36da8f05bffbb231b30d787e2dcb4d5ddd356c4fcc9f38467b5248
Sep 09 20:35:03.865141 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.865081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerStarted","Data":"a9c4321277c9c4fecc1595cb59d2f351b9d2c4490cfb1d25e0ba109ec3a11ab5"}
Sep 09 20:35:03.866747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.866716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerStarted","Data":"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"}
Sep 09 20:35:03.866968 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.866777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerStarted","Data":"03fadc86df36da8f05bffbb231b30d787e2dcb4d5ddd356c4fcc9f38467b5248"}
Sep 09 20:35:03.869214 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869190    2579 generic.go:356] "Generic (PLEG): container finished" podID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550" exitCode=0
Sep 09 20:35:03.869214 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869214    2579 generic.go:356] "Generic (PLEG): container finished" podID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65" exitCode=2
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869223    2579 generic.go:356] "Generic (PLEG): container finished" podID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d" exitCode=0
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerDied","Data":"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"}
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerDied","Data":"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"}
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerDied","Data":"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"}
Sep 09 20:35:03.869428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869367    2579 scope.go:117] "RemoveContainer" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"
Sep 09 20:35:03.869715 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.869372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4" event={"ID":"31109c8a-d680-40b1-a36e-6b95ec2afbe3","Type":"ContainerDied","Data":"0f8a4365ebc849d130eee9c7841e437ad5dea554acbe101ef70d480836c7cb61"}
Sep 09 20:35:03.877777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.877751    2579 scope.go:117] "RemoveContainer" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"
Sep 09 20:35:03.887373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.887350    2579 scope.go:117] "RemoveContainer" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"
Sep 09 20:35:03.900284 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.900260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:35:03.902906 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.902834    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-5b77679cf-kctj4"]
Sep 09 20:35:03.902976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.902952    2579 scope.go:117] "RemoveContainer" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"
Sep 09 20:35:03.914572 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.914543    2579 scope.go:117] "RemoveContainer" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"
Sep 09 20:35:03.914979 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:35:03.914959    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": container with ID starting with 2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550 not found: ID does not exist" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"
Sep 09 20:35:03.915081 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.914996    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"} err="failed to get container status \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": rpc error: code = NotFound desc = could not find container \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": container with ID starting with 2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550 not found: ID does not exist"
Sep 09 20:35:03.915081 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.915022    2579 scope.go:117] "RemoveContainer" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"
Sep 09 20:35:03.915372 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:35:03.915350    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": container with ID starting with b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65 not found: ID does not exist" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"
Sep 09 20:35:03.915439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.915380    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"} err="failed to get container status \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": rpc error: code = NotFound desc = could not find container \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": container with ID starting with b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65 not found: ID does not exist"
Sep 09 20:35:03.915439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.915397    2579 scope.go:117] "RemoveContainer" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"
Sep 09 20:35:03.915705 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:35:03.915672    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": container with ID starting with aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d not found: ID does not exist" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"
Sep 09 20:35:03.915823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.915700    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"} err="failed to get container status \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": rpc error: code = NotFound desc = could not find container \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": container with ID starting with aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d not found: ID does not exist"
Sep 09 20:35:03.915823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.915721    2579 scope.go:117] "RemoveContainer" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"
Sep 09 20:35:03.916044 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:35:03.916010    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": container with ID starting with 3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0 not found: ID does not exist" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"
Sep 09 20:35:03.916515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916049    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"} err="failed to get container status \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": rpc error: code = NotFound desc = could not find container \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": container with ID starting with 3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0 not found: ID does not exist"
Sep 09 20:35:03.916515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916071    2579 scope.go:117] "RemoveContainer" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"
Sep 09 20:35:03.916515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916386    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"} err="failed to get container status \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": rpc error: code = NotFound desc = could not find container \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": container with ID starting with 2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550 not found: ID does not exist"
Sep 09 20:35:03.916515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916405    2579 scope.go:117] "RemoveContainer" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"
Sep 09 20:35:03.916771 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916638    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"} err="failed to get container status \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": rpc error: code = NotFound desc = could not find container \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": container with ID starting with b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65 not found: ID does not exist"
Sep 09 20:35:03.916771 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916656    2579 scope.go:117] "RemoveContainer" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"
Sep 09 20:35:03.916948 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916920    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"} err="failed to get container status \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": rpc error: code = NotFound desc = could not find container \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": container with ID starting with aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d not found: ID does not exist"
Sep 09 20:35:03.917015 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.916950    2579 scope.go:117] "RemoveContainer" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"
Sep 09 20:35:03.917253 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"} err="failed to get container status \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": rpc error: code = NotFound desc = could not find container \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": container with ID starting with 3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0 not found: ID does not exist"
Sep 09 20:35:03.917352 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917253    2579 scope.go:117] "RemoveContainer" containerID="2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"
Sep 09 20:35:03.917553 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917532    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550"} err="failed to get container status \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": rpc error: code = NotFound desc = could not find container \"2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550\": container with ID starting with 2de0a36c1999f3c822c70fcc2d8ad43e26949602d815f3ab1a6e5b975d9a3550 not found: ID does not exist"
Sep 09 20:35:03.917626 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917556    2579 scope.go:117] "RemoveContainer" containerID="b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"
Sep 09 20:35:03.917857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917835    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65"} err="failed to get container status \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": rpc error: code = NotFound desc = could not find container \"b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65\": container with ID starting with b0d47f4669afbeecebeb99211f7df63879c6bbf43cbd87e66a25a86cb5cc7d65 not found: ID does not exist"
Sep 09 20:35:03.917946 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.917858    2579 scope.go:117] "RemoveContainer" containerID="aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"
Sep 09 20:35:03.918170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.918147    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d"} err="failed to get container status \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": rpc error: code = NotFound desc = could not find container \"aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d\": container with ID starting with aa22b6e1bd86a1760298cc5762c6008652a3dc2fce118ab68e21961297fc4a5d not found: ID does not exist"
Sep 09 20:35:03.918170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.918170    2579 scope.go:117] "RemoveContainer" containerID="3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"
Sep 09 20:35:03.918961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:03.918931    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0"} err="failed to get container status \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": rpc error: code = NotFound desc = could not find container \"3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0\": container with ID starting with 3cead1496730bf0f9ec8c1a31e53925764057f2e942f35caa310707011313bb0 not found: ID does not exist"
Sep 09 20:35:04.878479 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:04.878433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerStarted","Data":"24f6da10f9fad7cbc795016535e982b0aac5c3cc4936aa63e58cc55ccb5e68f8"}
Sep 09 20:35:04.899804 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:04.899752    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podStartSLOduration=2.376713755 podStartE2EDuration="2m15.899733763s" totalImagesPullingTime="2m9.514865642s" totalInitContainerRuntime="4.008154366s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:04.898224653 +0000 UTC m=+1437.734539117" watchObservedRunningTime="2026-09-09 20:35:04.899733763 +0000 UTC m=+1437.736048226"
Sep 09 20:35:05.606523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:05.606481    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:05.606985 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:05.606962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:05.611922 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:05.611886    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" path="/var/lib/kubelet/pods/31109c8a-d680-40b1-a36e-6b95ec2afbe3/volumes"
Sep 09 20:35:05.884565 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:05.884480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:35:05.884565 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:05.884538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:35:11.895274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:11.895239    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:35:15.607365 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:15.607315    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:15.607895 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:15.607867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:20.606492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:20.606444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:20.606956 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:20.606884    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:30.607391 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:30.607341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:30.607848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:30.607824    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:32.996504 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:32.996463    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerID="480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713" exitCode=0
Sep 09 20:35:32.996891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:32.996515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerDied","Data":"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"}
Sep 09 20:35:40.606824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:40.606767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:40.607362 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:40.607334    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:41.976165 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:41.976111    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:35:50.607327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:50.607275    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:35:50.607884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:50.607775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:35:51.898249 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:51.898215    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:35:59.672014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:59.671969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:35:59.672550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:59.672292    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container" containerID="cri-o://a9c4321277c9c4fecc1595cb59d2f351b9d2c4490cfb1d25e0ba109ec3a11ab5" gracePeriod=30
Sep 09 20:35:59.672550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:35:59.672485    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy" containerID="cri-o://24f6da10f9fad7cbc795016535e982b0aac5c3cc4936aa63e58cc55ccb5e68f8" gracePeriod=30
Sep 09 20:36:00.135250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:00.135153    2579 generic.go:356] "Generic (PLEG): container finished" podID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerID="24f6da10f9fad7cbc795016535e982b0aac5c3cc4936aa63e58cc55ccb5e68f8" exitCode=2
Sep 09 20:36:00.135420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:00.135238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerDied","Data":"24f6da10f9fad7cbc795016535e982b0aac5c3cc4936aa63e58cc55ccb5e68f8"}
Sep 09 20:36:00.607142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:00.607073    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:00.607620 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:00.607587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:01.890484 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:01.890429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 09 20:36:02.937349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:02.937286    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:36:04.883987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.883930    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884308    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884336    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884344    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884398    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="storage-initializer"
Sep 09 20:36:04.884493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884406    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="storage-initializer"
Sep 09 20:36:04.884927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884505    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kube-rbac-proxy"
Sep 09 20:36:04.884927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884519    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="agent"
Sep 09 20:36:04.884927 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.884534    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31109c8a-d680-40b1-a36e-6b95ec2afbe3" containerName="kserve-container"
Sep 09 20:36:04.887811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.887787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:04.890298 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.890274    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 09 20:36:04.890425 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.890278    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:36:04.896647 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.896623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:36:04.996020 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.995980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7lv\" (UniqueName: \"kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:04.996216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.996034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:04.996216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.996070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:04.996216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.996157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:04.996216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:04.996209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.096893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.096849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.096927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7lv\" (UniqueName: \"kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.096977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.097009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.097050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097344 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:36:05.097205    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 20:36:05.097344 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:36:05.097266    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls podName:9992b18c-5df6-4e79-9cce-da29ee41fdbb nodeName:}" failed. No retries permitted until 2026-09-09 20:36:05.597245697 +0000 UTC m=+1498.433560139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls") pod "isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 20:36:05.097444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.097368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.097817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.097880 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.097817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.107910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.107883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7lv\" (UniqueName: \"kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.601779 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.601741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.604822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.604792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:05.798685 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:05.798651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:06.085401 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:06.085370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:36:06.087444 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:36:06.087398    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9992b18c_5df6_4e79_9cce_da29ee41fdbb.slice/crio-ac431e70fa06379c1997530a1dbaffa0cf5057e775efe3f9d84c5e1a74a9b8fa WatchSource:0}: Error finding container ac431e70fa06379c1997530a1dbaffa0cf5057e775efe3f9d84c5e1a74a9b8fa: Status 404 returned error can't find the container with id ac431e70fa06379c1997530a1dbaffa0cf5057e775efe3f9d84c5e1a74a9b8fa
Sep 09 20:36:06.166353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:06.166314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerStarted","Data":"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"}
Sep 09 20:36:06.166488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:06.166400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerStarted","Data":"ac431e70fa06379c1997530a1dbaffa0cf5057e775efe3f9d84c5e1a74a9b8fa"}
Sep 09 20:36:06.890416 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:06.890375    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 09 20:36:08.180037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.179993    2579 generic.go:356] "Generic (PLEG): container finished" podID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerID="a9c4321277c9c4fecc1595cb59d2f351b9d2c4490cfb1d25e0ba109ec3a11ab5" exitCode=0
Sep 09 20:36:08.180495 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.180068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerDied","Data":"a9c4321277c9c4fecc1595cb59d2f351b9d2c4490cfb1d25e0ba109ec3a11ab5"}
Sep 09 20:36:08.247585 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.247537    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:36:08.325863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.325820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location\") pod \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") "
Sep 09 20:36:08.326062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.325898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls\") pod \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") "
Sep 09 20:36:08.326062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.325940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kz5b\" (UniqueName: \"kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b\") pod \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") "
Sep 09 20:36:08.326062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.325966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config\") pod \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") "
Sep 09 20:36:08.326062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.326028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert\") pod \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\" (UID: \"d8e62a33-91ed-451a-b7e1-7183cbd2e627\") "
Sep 09 20:36:08.326317 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.326208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8e62a33-91ed-451a-b7e1-7183cbd2e627" (UID: "d8e62a33-91ed-451a-b7e1-7183cbd2e627"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:08.326375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.326334    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:36:08.326535 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.326514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8e62a33-91ed-451a-b7e1-7183cbd2e627" (UID: "d8e62a33-91ed-451a-b7e1-7183cbd2e627"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:08.326728 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.326679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8e62a33-91ed-451a-b7e1-7183cbd2e627" (UID: "d8e62a33-91ed-451a-b7e1-7183cbd2e627"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:08.328691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.328659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b" (OuterVolumeSpecName: "kube-api-access-4kz5b") pod "d8e62a33-91ed-451a-b7e1-7183cbd2e627" (UID: "d8e62a33-91ed-451a-b7e1-7183cbd2e627"). InnerVolumeSpecName "kube-api-access-4kz5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:08.328864 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.328840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8e62a33-91ed-451a-b7e1-7183cbd2e627" (UID: "d8e62a33-91ed-451a-b7e1-7183cbd2e627"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:08.427429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.427331    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kz5b\" (UniqueName: \"kubernetes.io/projected/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-api-access-4kz5b\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:36:08.427429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.427375    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:36:08.427429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.427392    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e62a33-91ed-451a-b7e1-7183cbd2e627-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:36:08.427429 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:08.427405    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8e62a33-91ed-451a-b7e1-7183cbd2e627-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.188139 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.188082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw" event={"ID":"d8e62a33-91ed-451a-b7e1-7183cbd2e627","Type":"ContainerDied","Data":"3be653b1ef5aae83a43aea45b8406db64ea8d7a5756f1ca2a1af84aba67f28e8"}
Sep 09 20:36:09.188602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.188174    2579 scope.go:117] "RemoveContainer" containerID="24f6da10f9fad7cbc795016535e982b0aac5c3cc4936aa63e58cc55ccb5e68f8"
Sep 09 20:36:09.188602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.188108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"
Sep 09 20:36:09.198772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.198742    2579 scope.go:117] "RemoveContainer" containerID="a9c4321277c9c4fecc1595cb59d2f351b9d2c4490cfb1d25e0ba109ec3a11ab5"
Sep 09 20:36:09.206718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.206699    2579 scope.go:117] "RemoveContainer" containerID="205c6e89558cfc3377500421db879e8f7fe6caf40b3c08d104f814e6c3a399f4"
Sep 09 20:36:09.213532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.213509    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:36:09.218635 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.218612    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-fbwsw"]
Sep 09 20:36:09.612921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:09.612821    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" path="/var/lib/kubelet/pods/d8e62a33-91ed-451a-b7e1-7183cbd2e627/volumes"
Sep 09 20:36:10.606411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:10.606352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:10.606888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:10.606830    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:13.210765 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:13.210727    2579 generic.go:356] "Generic (PLEG): container finished" podID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerID="e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4" exitCode=0
Sep 09 20:36:13.211204 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:13.210771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerDied","Data":"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"}
Sep 09 20:36:14.218509 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:14.218469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerStarted","Data":"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"}
Sep 09 20:36:14.218935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:14.218564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerStarted","Data":"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"}
Sep 09 20:36:14.218935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:14.218830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:14.237958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:14.237902    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podStartSLOduration=3.624561163 podStartE2EDuration="10.237878387s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.613317224s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:14.236242182 +0000 UTC m=+1507.072556646" watchObservedRunningTime="2026-09-09 20:36:14.237878387 +0000 UTC m=+1507.074192850"
Sep 09 20:36:15.226201 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:15.226123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:15.227444 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:15.227409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:16.231223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:16.231186    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:20.606741 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:20.606687    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:20.607570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:20.607519    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:21.237753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:21.237720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:21.238375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:21.238335    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:30.606498 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:30.606400    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:30.606976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:30.606889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:31.238544 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:31.238500    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:40.606372 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:40.606320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:40.606938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:40.606910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:41.239385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:41.239349    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:36:47.147838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:47.147804    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:36:47.148346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:47.148120    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" containerID="cri-o://e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3" gracePeriod=30
Sep 09 20:36:47.148441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:47.148223    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" containerID="cri-o://4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2" gracePeriod=30
Sep 09 20:36:50.606418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:50.606365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:36:50.606871 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:50.606840    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:36:51.231636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:51.231587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:36:51.238677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:51.238635    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:36:52.309549 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.309506    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="storage-initializer"
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310081    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="storage-initializer"
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310126    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy"
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310135    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy"
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310148    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container"
Sep 09 20:36:52.310167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container"
Sep 09 20:36:52.310527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310260    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kserve-container"
Sep 09 20:36:52.310527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.310276    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e62a33-91ed-451a-b7e1-7183cbd2e627" containerName="kube-rbac-proxy"
Sep 09 20:36:52.314515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.314489    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.316841 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.316808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 09 20:36:52.317004 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.316863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:36:52.325883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.325853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:36:52.476439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.476399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.476658 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.476474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.476658 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.476498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.476658 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.476594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9z8m\" (UniqueName: \"kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.476849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.476704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9z8m\" (UniqueName: \"kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578407 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:36:52.578311    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 09 20:36:52.578407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578407 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:36:52.578377    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls podName:5b419575-e689-470a-939e-4eb2d1e98ddb nodeName:}" failed. No retries permitted until 2026-09-09 20:36:53.078357974 +0000 UTC m=+1545.914672416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 09 20:36:52.578748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.578947 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.578927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.579118 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.579075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:52.586985 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:52.586952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9z8m\" (UniqueName: \"kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:53.084335 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:53.084294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:53.087217 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:53.087187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:53.228157 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:53.228116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:36:56.231823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:36:56.231773    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:00.606456 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:00.606417    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:00.859273 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:00.606969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:01.231916 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:01.231869    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:01.232151 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:01.232024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:37:01.238304 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:01.238269    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:06.231887 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:06.231831    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:10.607195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:10.607144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:10.607689 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:10.607581    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:11.231704 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:11.231660    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:11.239245 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:11.239202    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:11.239434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:11.239365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:37:16.232195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:16.232140    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:20.606385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:20.606330    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:20.607246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:20.607211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:21.232532 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:21.232481    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:21.238884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:21.238845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:26.232331 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:26.232275    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:30.607135 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:30.607065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:30.607681 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:30.607655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:31.232318 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:31.232270    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 09 20:37:31.238702 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:31.238656    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 09 20:37:33.138219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.138190    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:37:33.263826 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.263786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location\") pod \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") "
Sep 09 20:37:33.264039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.263881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert\") pod \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") "
Sep 09 20:37:33.264039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.263924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz7lv\" (UniqueName: \"kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv\") pod \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") "
Sep 09 20:37:33.264039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.263975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") pod \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") "
Sep 09 20:37:33.264039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.264008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config\") pod \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\" (UID: \"9992b18c-5df6-4e79-9cce-da29ee41fdbb\") "
Sep 09 20:37:33.264449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.264408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9992b18c-5df6-4e79-9cce-da29ee41fdbb" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:33.264534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.264508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9992b18c-5df6-4e79-9cce-da29ee41fdbb" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:33.266384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.266350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv" (OuterVolumeSpecName: "kube-api-access-cz7lv") pod "9992b18c-5df6-4e79-9cce-da29ee41fdbb" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb"). InnerVolumeSpecName "kube-api-access-cz7lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:33.266486 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.266380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9992b18c-5df6-4e79-9cce-da29ee41fdbb" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:33.267878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.267856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9992b18c-5df6-4e79-9cce-da29ee41fdbb" (UID: "9992b18c-5df6-4e79-9cce-da29ee41fdbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:33.330313 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.330282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:37:33.332423 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:37:33.332383    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b419575_e689_470a_939e_4eb2d1e98ddb.slice/crio-dd7ba8f63b5ea176f2d26b9efc0b1eff4f9de7c99a666f4efc151de306531247 WatchSource:0}: Error finding container dd7ba8f63b5ea176f2d26b9efc0b1eff4f9de7c99a666f4efc151de306531247: Status 404 returned error can't find the container with id dd7ba8f63b5ea176f2d26b9efc0b1eff4f9de7c99a666f4efc151de306531247
Sep 09 20:37:33.365601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.365572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz7lv\" (UniqueName: \"kubernetes.io/projected/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-api-access-cz7lv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:37:33.365699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.365604    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9992b18c-5df6-4e79-9cce-da29ee41fdbb-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:37:33.365699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.365623    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:37:33.365699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.365635    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9992b18c-5df6-4e79-9cce-da29ee41fdbb-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:37:33.365699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.365648    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9992b18c-5df6-4e79-9cce-da29ee41fdbb-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:37:33.634743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.634705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerStarted","Data":"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"}
Sep 09 20:37:33.634863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.634787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerStarted","Data":"dd7ba8f63b5ea176f2d26b9efc0b1eff4f9de7c99a666f4efc151de306531247"}
Sep 09 20:37:33.637226 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637197    2579 generic.go:356] "Generic (PLEG): container finished" podID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerID="4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2" exitCode=2
Sep 09 20:37:33.637323 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637229    2579 generic.go:356] "Generic (PLEG): container finished" podID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerID="e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3" exitCode=0
Sep 09 20:37:33.637323 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637260    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"
Sep 09 20:37:33.637402 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerDied","Data":"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"}
Sep 09 20:37:33.637437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerDied","Data":"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"}
Sep 09 20:37:33.637494 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr" event={"ID":"9992b18c-5df6-4e79-9cce-da29ee41fdbb","Type":"ContainerDied","Data":"ac431e70fa06379c1997530a1dbaffa0cf5057e775efe3f9d84c5e1a74a9b8fa"}
Sep 09 20:37:33.637494 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.637482    2579 scope.go:117] "RemoveContainer" containerID="4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"
Sep 09 20:37:33.646233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.646214    2579 scope.go:117] "RemoveContainer" containerID="e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"
Sep 09 20:37:33.661604 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.661579    2579 scope.go:117] "RemoveContainer" containerID="e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"
Sep 09 20:37:33.665285 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.665260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:37:33.670156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.670129    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-7df7c497c6-4hwkr"]
Sep 09 20:37:33.677836 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.677816    2579 scope.go:117] "RemoveContainer" containerID="4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"
Sep 09 20:37:33.678291 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:37:33.678265    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2\": container with ID starting with 4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2 not found: ID does not exist" containerID="4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"
Sep 09 20:37:33.678382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"} err="failed to get container status \"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2\": rpc error: code = NotFound desc = could not find container \"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2\": container with ID starting with 4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2 not found: ID does not exist"
Sep 09 20:37:33.678382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678319    2579 scope.go:117] "RemoveContainer" containerID="e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"
Sep 09 20:37:33.678628 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:37:33.678610    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3\": container with ID starting with e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3 not found: ID does not exist" containerID="e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"
Sep 09 20:37:33.678678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678636    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"} err="failed to get container status \"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3\": rpc error: code = NotFound desc = could not find container \"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3\": container with ID starting with e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3 not found: ID does not exist"
Sep 09 20:37:33.678678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678659    2579 scope.go:117] "RemoveContainer" containerID="e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"
Sep 09 20:37:33.678942 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:37:33.678925    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4\": container with ID starting with e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4 not found: ID does not exist" containerID="e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"
Sep 09 20:37:33.679019 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678946    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"} err="failed to get container status \"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4\": rpc error: code = NotFound desc = could not find container \"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4\": container with ID starting with e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4 not found: ID does not exist"
Sep 09 20:37:33.679019 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.678961    2579 scope.go:117] "RemoveContainer" containerID="4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"
Sep 09 20:37:33.679240 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.679224    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2"} err="failed to get container status \"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2\": rpc error: code = NotFound desc = could not find container \"4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2\": container with ID starting with 4e61e329ec4e87d805ea35684bfb80d1fd98547b21e0d00024aaa9d4152a0ec2 not found: ID does not exist"
Sep 09 20:37:33.679303 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.679240    2579 scope.go:117] "RemoveContainer" containerID="e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"
Sep 09 20:37:33.679508 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.679490    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3"} err="failed to get container status \"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3\": rpc error: code = NotFound desc = could not find container \"e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3\": container with ID starting with e8545d3cb4fc0407d8f56bbeddf5c60b7bdc18c78c85108c351c6c46578e34c3 not found: ID does not exist"
Sep 09 20:37:33.679569 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.679509    2579 scope.go:117] "RemoveContainer" containerID="e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"
Sep 09 20:37:33.679764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:33.679732    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4"} err="failed to get container status \"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4\": rpc error: code = NotFound desc = could not find container \"e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4\": container with ID starting with e46be77aa98140bf99b5a03be855260846c4c67cca11c3795e2bb6e2919f04b4 not found: ID does not exist"
Sep 09 20:37:34.644450 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:34.644409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerStarted","Data":"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"}
Sep 09 20:37:34.645006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:34.644496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerStarted","Data":"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"}
Sep 09 20:37:34.645006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:34.644578    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:37:34.666735 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:34.666686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podStartSLOduration=35.629124584 podStartE2EDuration="3m5.666669204s" totalImagesPullingTime="2m0.569271127s" totalInitContainerRuntime="29.468273493s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:34.665598702 +0000 UTC m=+1587.501913168" watchObservedRunningTime="2026-09-09 20:37:34.666669204 +0000 UTC m=+1587.502983666"
Sep 09 20:37:35.610893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:35.610858    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" path="/var/lib/kubelet/pods/9992b18c-5df6-4e79-9cce-da29ee41fdbb/volumes"
Sep 09 20:37:35.650753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:35.650726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:37:35.652306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:35.652280    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:37:36.656287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:36.656243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:37:38.672606 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:38.672561    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerID="ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b" exitCode=0
Sep 09 20:37:38.672999 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:38.672629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerDied","Data":"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"}
Sep 09 20:37:39.680641 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:39.680602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerStarted","Data":"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"}
Sep 09 20:37:39.681269 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:39.680703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerStarted","Data":"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"}
Sep 09 20:37:39.681269 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:39.680937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:37:39.681269 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:39.680988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:37:39.712984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:39.712929    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" podStartSLOduration=43.105442378 podStartE2EDuration="47.712914188s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.60747181s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:39.71218358 +0000 UTC m=+1592.548539315" watchObservedRunningTime="2026-09-09 20:37:39.712914188 +0000 UTC m=+1592.549228651"
Sep 09 20:37:40.606620 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:40.606571    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:40.607142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:40.607118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:41.661437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:41.661393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:37:41.662017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:41.661990    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 09 20:37:45.692203 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:45.692171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:37:49.606379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:49.606322    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:49.606797 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:49.606771    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:37:51.662755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:51.662724    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:37:59.606478 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:59.606371    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:37:59.606934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:37:59.606786    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:06.551108 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.551051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:38:06.551620 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.551338    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container" containerID="cri-o://088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66" gracePeriod=30
Sep 09 20:38:06.551620 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.551354    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kube-rbac-proxy" containerID="cri-o://2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c" gracePeriod=30
Sep 09 20:38:06.656987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.656946    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 09 20:38:06.800993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.800956    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerID="2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c" exitCode=2
Sep 09 20:38:06.800993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:06.800982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerDied","Data":"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"}
Sep 09 20:38:08.237759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.237683    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:38:08.284700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.284666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location\") pod \"0ab97367-a1c2-4195-a6f6-55d21692e01e\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") "
Sep 09 20:38:08.284885 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.284724    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert\") pod \"0ab97367-a1c2-4195-a6f6-55d21692e01e\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") "
Sep 09 20:38:08.284885 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.284767    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7k67\" (UniqueName: \"kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67\") pod \"0ab97367-a1c2-4195-a6f6-55d21692e01e\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") "
Sep 09 20:38:08.284885 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.284789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config\") pod \"0ab97367-a1c2-4195-a6f6-55d21692e01e\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") "
Sep 09 20:38:08.284885 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.284833    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls\") pod \"0ab97367-a1c2-4195-a6f6-55d21692e01e\" (UID: \"0ab97367-a1c2-4195-a6f6-55d21692e01e\") "
Sep 09 20:38:08.285153 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.285119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ab97367-a1c2-4195-a6f6-55d21692e01e" (UID: "0ab97367-a1c2-4195-a6f6-55d21692e01e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:08.285279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.285239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ab97367-a1c2-4195-a6f6-55d21692e01e" (UID: "0ab97367-a1c2-4195-a6f6-55d21692e01e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:08.287207 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.287176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67" (OuterVolumeSpecName: "kube-api-access-w7k67") pod "0ab97367-a1c2-4195-a6f6-55d21692e01e" (UID: "0ab97367-a1c2-4195-a6f6-55d21692e01e"). InnerVolumeSpecName "kube-api-access-w7k67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:08.287633 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.287613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ab97367-a1c2-4195-a6f6-55d21692e01e" (UID: "0ab97367-a1c2-4195-a6f6-55d21692e01e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:08.289134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.289109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ab97367-a1c2-4195-a6f6-55d21692e01e" (UID: "0ab97367-a1c2-4195-a6f6-55d21692e01e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:08.385547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.385507    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ab97367-a1c2-4195-a6f6-55d21692e01e-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:08.385547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.385539    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:08.385547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.385548    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7k67\" (UniqueName: \"kubernetes.io/projected/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-api-access-w7k67\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:08.385792 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.385558    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ab97367-a1c2-4195-a6f6-55d21692e01e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:08.385792 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.385571    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ab97367-a1c2-4195-a6f6-55d21692e01e-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:08.811672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.811629    2579 generic.go:356] "Generic (PLEG): container finished" podID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerID="088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66" exitCode=0
Sep 09 20:38:08.811853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.811689    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"
Sep 09 20:38:08.811853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.811717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerDied","Data":"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"}
Sep 09 20:38:08.811853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.811781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt" event={"ID":"0ab97367-a1c2-4195-a6f6-55d21692e01e","Type":"ContainerDied","Data":"03fadc86df36da8f05bffbb231b30d787e2dcb4d5ddd356c4fcc9f38467b5248"}
Sep 09 20:38:08.811853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.811810    2579 scope.go:117] "RemoveContainer" containerID="2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"
Sep 09 20:38:08.820980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.820948    2579 scope.go:117] "RemoveContainer" containerID="088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"
Sep 09 20:38:08.829657 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.829636    2579 scope.go:117] "RemoveContainer" containerID="480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"
Sep 09 20:38:08.834397 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.834338    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:38:08.838255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.838226    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jb8tt"]
Sep 09 20:38:08.839436 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.839415    2579 scope.go:117] "RemoveContainer" containerID="2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"
Sep 09 20:38:08.839815 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:08.839791    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c\": container with ID starting with 2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c not found: ID does not exist" containerID="2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"
Sep 09 20:38:08.839911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.839830    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c"} err="failed to get container status \"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c\": rpc error: code = NotFound desc = could not find container \"2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c\": container with ID starting with 2ed4872dfc7b2a8b985e770ba197ba2a1880fe61a10525d0f304eaaf09dd825c not found: ID does not exist"
Sep 09 20:38:08.839911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.839859    2579 scope.go:117] "RemoveContainer" containerID="088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"
Sep 09 20:38:08.840211 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:08.840191    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66\": container with ID starting with 088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66 not found: ID does not exist" containerID="088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"
Sep 09 20:38:08.840303 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.840215    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66"} err="failed to get container status \"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66\": rpc error: code = NotFound desc = could not find container \"088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66\": container with ID starting with 088262559c09c0836c82a2f5926ac81dc1538b910530c8bf1a5991a4dfdeda66 not found: ID does not exist"
Sep 09 20:38:08.840303 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.840231    2579 scope.go:117] "RemoveContainer" containerID="480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"
Sep 09 20:38:08.840497 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:08.840479    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713\": container with ID starting with 480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713 not found: ID does not exist" containerID="480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"
Sep 09 20:38:08.840550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:08.840506    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713"} err="failed to get container status \"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713\": rpc error: code = NotFound desc = could not find container \"480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713\": container with ID starting with 480ec8ce377e098ec06ee8f698508647a1a920ab6dad1b496a283f89c7bfb713 not found: ID does not exist"
Sep 09 20:38:09.606298 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:09.606251    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:09.606788 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:09.606737    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:09.610923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:09.610894    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" path="/var/lib/kubelet/pods/0ab97367-a1c2-4195-a6f6-55d21692e01e/volumes"
Sep 09 20:38:11.713703 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.713672    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.713999    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714012    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714034    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="storage-initializer"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714039    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="storage-initializer"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714055    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714063    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714068    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714075    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="storage-initializer"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714080    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="storage-initializer"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714099    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714104    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714151    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714159    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kube-rbac-proxy"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab97367-a1c2-4195-a6f6-55d21692e01e" containerName="kserve-container"
Sep 09 20:38:11.714206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.714172    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9992b18c-5df6-4e79-9cce-da29ee41fdbb" containerName="kserve-container"
Sep 09 20:38:11.717450 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.717429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.719782 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.719765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:38:11.719888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.719770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 09 20:38:11.727145 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.727118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:38:11.816270 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.816236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.816270 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.816282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.816520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.816318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.816520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.816355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.816520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.816384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2jr\" (UniqueName: \"kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2jr\" (UniqueName: \"kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917561 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.917973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.917958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.919861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.919838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:11.925765 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:11.925742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2jr\" (UniqueName: \"kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-twxfb\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:12.028389 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:12.028286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:12.305244 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:12.305214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:38:12.306451 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:38:12.306423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b33e023_bd06_4aee_b74c_4b83611083df.slice/crio-ab57222df71040f6d199cd26a90bb8220f009c2e6f5d88288a91a47ce36acf23 WatchSource:0}: Error finding container ab57222df71040f6d199cd26a90bb8220f009c2e6f5d88288a91a47ce36acf23: Status 404 returned error can't find the container with id ab57222df71040f6d199cd26a90bb8220f009c2e6f5d88288a91a47ce36acf23
Sep 09 20:38:12.831394 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:12.831357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerStarted","Data":"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"}
Sep 09 20:38:12.831810 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:12.831415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerStarted","Data":"ab57222df71040f6d199cd26a90bb8220f009c2e6f5d88288a91a47ce36acf23"}
Sep 09 20:38:15.776188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:15.776148    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:38:19.606932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:19.606882    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:19.607464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:19.607390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:25.695068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:25.695039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:38:29.606732 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:29.606684    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:29.607264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:29.607241    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:32.399812 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:32.399779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:38:32.400353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:32.400008    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kserve-container" containerID="cri-o://429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b" gracePeriod=30
Sep 09 20:38:32.400353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:32.400080    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kube-rbac-proxy" containerID="cri-o://5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9" gracePeriod=30
Sep 09 20:38:32.917559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:32.917460    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerID="5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9" exitCode=2
Sep 09 20:38:32.917559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:32.917529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerDied","Data":"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"}
Sep 09 20:38:35.687549 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:35.687504    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 09 20:38:37.564523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.564485    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:38:37.569625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.569601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.572232 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.572208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 09 20:38:37.572380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.572293    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:38:37.578278 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.578245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:38:37.741414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.741376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.741414 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.741414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.741688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.741520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgxj\" (UniqueName: \"kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.741688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.741577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.741688 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.741619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.842502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.842405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.842502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.842447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.842502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.842500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgxj\" (UniqueName: \"kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.842787 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.842529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.842787 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.842560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.843082 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.843048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.843296 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.843275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.843351 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.843292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.845122 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.845103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.850873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.850850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgxj\" (UniqueName: \"kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj\") pod \"isvc-sklearn-v2-predictor-5b94c8df76-96hrw\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:37.882350 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:37.882319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:38.153878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:38.153847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:38:38.156810 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:38:38.156782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f120bd4_9753_4fb9_9273_d7655aee0df8.slice/crio-cc91f2a0518205676d3218dc59521f83303363678ec9eabb11583d092c7dd733 WatchSource:0}: Error finding container cc91f2a0518205676d3218dc59521f83303363678ec9eabb11583d092c7dd733: Status 404 returned error can't find the container with id cc91f2a0518205676d3218dc59521f83303363678ec9eabb11583d092c7dd733
Sep 09 20:38:38.946634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:38.946596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerStarted","Data":"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"}
Sep 09 20:38:38.947043 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:38.946660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerStarted","Data":"cc91f2a0518205676d3218dc59521f83303363678ec9eabb11583d092c7dd733"}
Sep 09 20:38:39.606293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:39.606248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:39.606716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:39.606692    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:40.144554 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.144523    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:38:40.263359 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert\") pod \"5b419575-e689-470a-939e-4eb2d1e98ddb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") "
Sep 09 20:38:40.263359 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location\") pod \"5b419575-e689-470a-939e-4eb2d1e98ddb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") "
Sep 09 20:38:40.263611 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config\") pod \"5b419575-e689-470a-939e-4eb2d1e98ddb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") "
Sep 09 20:38:40.263611 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9z8m\" (UniqueName: \"kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m\") pod \"5b419575-e689-470a-939e-4eb2d1e98ddb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") "
Sep 09 20:38:40.263611 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") pod \"5b419575-e689-470a-939e-4eb2d1e98ddb\" (UID: \"5b419575-e689-470a-939e-4eb2d1e98ddb\") "
Sep 09 20:38:40.263780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b419575-e689-470a-939e-4eb2d1e98ddb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:40.263780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263736    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b419575-e689-470a-939e-4eb2d1e98ddb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:40.263780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.263768    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b419575-e689-470a-939e-4eb2d1e98ddb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:40.265691 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.265657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m" (OuterVolumeSpecName: "kube-api-access-n9z8m") pod "5b419575-e689-470a-939e-4eb2d1e98ddb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb"). InnerVolumeSpecName "kube-api-access-n9z8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:40.265802 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.265752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b419575-e689-470a-939e-4eb2d1e98ddb" (UID: "5b419575-e689-470a-939e-4eb2d1e98ddb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:40.364330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.364293    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.364330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.364323    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9z8m\" (UniqueName: \"kubernetes.io/projected/5b419575-e689-470a-939e-4eb2d1e98ddb-kube-api-access-n9z8m\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.364330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.364333    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b419575-e689-470a-939e-4eb2d1e98ddb-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.364572 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.364344    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b419575-e689-470a-939e-4eb2d1e98ddb-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.364572 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.364352    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b419575-e689-470a-939e-4eb2d1e98ddb-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.957282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.957243    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerID="429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b" exitCode=0
Sep 09 20:38:40.957483 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.957301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"
Sep 09 20:38:40.957483 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.957296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerDied","Data":"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"}
Sep 09 20:38:40.957483 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.957380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb" event={"ID":"5b419575-e689-470a-939e-4eb2d1e98ddb","Type":"ContainerDied","Data":"dd7ba8f63b5ea176f2d26b9efc0b1eff4f9de7c99a666f4efc151de306531247"}
Sep 09 20:38:40.957483 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.957418    2579 scope.go:117] "RemoveContainer" containerID="5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"
Sep 09 20:38:40.970356 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.970330    2579 scope.go:117] "RemoveContainer" containerID="429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"
Sep 09 20:38:40.978189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.978167    2579 scope.go:117] "RemoveContainer" containerID="ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"
Sep 09 20:38:40.980848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.980822    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:38:40.983955 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.983929    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2w7hb"]
Sep 09 20:38:40.986846 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.986822    2579 scope.go:117] "RemoveContainer" containerID="5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"
Sep 09 20:38:40.987357 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:40.987338    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9\": container with ID starting with 5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9 not found: ID does not exist" containerID="5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"
Sep 09 20:38:40.987428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.987366    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9"} err="failed to get container status \"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9\": rpc error: code = NotFound desc = could not find container \"5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9\": container with ID starting with 5b55940947e26e781e8f48e1af257abfc083922f31d150db4a5590c7493c2cc9 not found: ID does not exist"
Sep 09 20:38:40.987428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.987384    2579 scope.go:117] "RemoveContainer" containerID="429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"
Sep 09 20:38:40.987681 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:40.987663    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b\": container with ID starting with 429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b not found: ID does not exist" containerID="429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"
Sep 09 20:38:40.987723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.987688    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b"} err="failed to get container status \"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b\": rpc error: code = NotFound desc = could not find container \"429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b\": container with ID starting with 429749a2ad94142507fcbf29dfe94c960cc69e4a2f6840a8f071764ecd554c2b not found: ID does not exist"
Sep 09 20:38:40.987723 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.987705    2579 scope.go:117] "RemoveContainer" containerID="ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"
Sep 09 20:38:40.987936 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:38:40.987919    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b\": container with ID starting with ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b not found: ID does not exist" containerID="ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"
Sep 09 20:38:40.987982 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:40.987940    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b"} err="failed to get container status \"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b\": rpc error: code = NotFound desc = could not find container \"ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b\": container with ID starting with ebfbea464124714f3fe77df0ab2da31bd4d6ceb8464d1162573cd71a0ff42c3b not found: ID does not exist"
Sep 09 20:38:41.610481 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:41.610450    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" path="/var/lib/kubelet/pods/5b419575-e689-470a-939e-4eb2d1e98ddb/volumes"
Sep 09 20:38:42.969413 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:42.969376    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerID="fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67" exitCode=0
Sep 09 20:38:42.969869 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:42.969455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerDied","Data":"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"}
Sep 09 20:38:42.971217 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:42.971138    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b33e023-bd06-4aee-b74c-4b83611083df" containerID="9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc" exitCode=0
Sep 09 20:38:42.971217 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:42.971179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerDied","Data":"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"}
Sep 09 20:38:43.978606 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.978572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerStarted","Data":"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"}
Sep 09 20:38:43.979145 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.978645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerStarted","Data":"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"}
Sep 09 20:38:43.979145 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.978890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:43.981021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.980995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerStarted","Data":"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"}
Sep 09 20:38:43.981164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.981048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerStarted","Data":"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"}
Sep 09 20:38:43.981263 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.981252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:43.999293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:43.999231    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podStartSLOduration=2.8946473729999997 podStartE2EDuration="6.999214675s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.104567302s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:43.996636364 +0000 UTC m=+1656.832950828" watchObservedRunningTime="2026-09-09 20:38:43.999214675 +0000 UTC m=+1656.835529137"
Sep 09 20:38:44.014806 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:44.014753    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" podStartSLOduration=3.398276949 podStartE2EDuration="33.014732803s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.616455854s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:44.013505954 +0000 UTC m=+1656.849820436" watchObservedRunningTime="2026-09-09 20:38:44.014732803 +0000 UTC m=+1656.851047266"
Sep 09 20:38:44.988069 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:44.988028    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:44.988577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:44.988119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:44.989296 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:44.989266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:38:45.992049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:45.992007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:38:49.606499 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:49.606444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:49.606920 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:49.606896    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:38:50.997438 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:50.997409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:38:50.997977 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:50.997953    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:38:50.998049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:50.997972    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:38:59.606495 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:59.606456    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:38:59.606968 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:38:59.606941    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:00.606319 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:00.606271    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:00.606623 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:00.606599    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:00.998808 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:00.998769    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:39:10.606193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:10.606151    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:10.606763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:10.606559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:10.998068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:10.998027    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:39:20.606849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:20.606796    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:20.607321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:20.607296    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:20.997995 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:20.997958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:39:30.606514 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:30.606408    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:30.606892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:30.606808    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:30.998697 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:30.998654    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:39:40.606488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:40.606436    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:40.606959 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:40.606929    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:40.998415 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:40.998375    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:39:50.606213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:50.606169    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:39:50.606726 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:50.606700    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:39:50.998768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:39:50.998729    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:40:00.606218 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:00.606153    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:00.606622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:00.606590    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:00.999482 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:00.999445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:40:07.757281 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:07.757243    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:40:07.757748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:07.757536    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" containerID="cri-o://8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220" gracePeriod=30
Sep 09 20:40:07.757748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:07.757613    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kube-rbac-proxy" containerID="cri-o://6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6" gracePeriod=30
Sep 09 20:40:08.344597 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:08.344558    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerID="6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6" exitCode=2
Sep 09 20:40:08.344771 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:08.344625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerDied","Data":"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"}
Sep 09 20:40:08.606117 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:08.605991    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:08.606366 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:08.606341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:10.992346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:10.992301    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 09 20:40:10.998449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:10.998418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 09 20:40:12.696599 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.696572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:40:12.759990 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.759897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls\") pod \"9f120bd4-9753-4fb9-9273-d7655aee0df8\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") "
Sep 09 20:40:12.760178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.759999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert\") pod \"9f120bd4-9753-4fb9-9273-d7655aee0df8\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") "
Sep 09 20:40:12.760178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxgxj\" (UniqueName: \"kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj\") pod \"9f120bd4-9753-4fb9-9273-d7655aee0df8\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") "
Sep 09 20:40:12.760178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config\") pod \"9f120bd4-9753-4fb9-9273-d7655aee0df8\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") "
Sep 09 20:40:12.760178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760149    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location\") pod \"9f120bd4-9753-4fb9-9273-d7655aee0df8\" (UID: \"9f120bd4-9753-4fb9-9273-d7655aee0df8\") "
Sep 09 20:40:12.760449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f120bd4-9753-4fb9-9273-d7655aee0df8" (UID: "9f120bd4-9753-4fb9-9273-d7655aee0df8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:12.760588 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f120bd4-9753-4fb9-9273-d7655aee0df8" (UID: "9f120bd4-9753-4fb9-9273-d7655aee0df8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:12.760642 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.760585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f120bd4-9753-4fb9-9273-d7655aee0df8" (UID: "9f120bd4-9753-4fb9-9273-d7655aee0df8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:12.762358 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.762336    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f120bd4-9753-4fb9-9273-d7655aee0df8" (UID: "9f120bd4-9753-4fb9-9273-d7655aee0df8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:12.762442 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.762366    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj" (OuterVolumeSpecName: "kube-api-access-pxgxj") pod "9f120bd4-9753-4fb9-9273-d7655aee0df8" (UID: "9f120bd4-9753-4fb9-9273-d7655aee0df8"). InnerVolumeSpecName "kube-api-access-pxgxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:12.860775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.860743    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:12.860775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.860773    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgxj\" (UniqueName: \"kubernetes.io/projected/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-api-access-pxgxj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:12.860775 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.860784    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f120bd4-9753-4fb9-9273-d7655aee0df8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:12.861000 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.860793    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f120bd4-9753-4fb9-9273-d7655aee0df8-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:12.861000 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.860802    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f120bd4-9753-4fb9-9273-d7655aee0df8-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:12.924829 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.924790    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:40:12.925167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925152    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kube-rbac-proxy"
Sep 09 20:40:12.925219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kube-rbac-proxy"
Sep 09 20:40:12.925219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="storage-initializer"
Sep 09 20:40:12.925219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925198    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="storage-initializer"
Sep 09 20:40:12.925219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="storage-initializer"
Sep 09 20:40:12.925219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925220    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="storage-initializer"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925227    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kube-rbac-proxy"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925232    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kube-rbac-proxy"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925245    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925250    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925261    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925332    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925343    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kserve-container"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b419575-e689-470a-939e-4eb2d1e98ddb" containerName="kube-rbac-proxy"
Sep 09 20:40:12.925380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.925363    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerName="kube-rbac-proxy"
Sep 09 20:40:12.928809 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.928793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:12.931251 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.931231    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 09 20:40:12.931251 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.931239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 09 20:40:12.938178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:12.938154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:40:13.063134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.063023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.063134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.063071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.063333 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.063164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.063333 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.063191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n79p\" (UniqueName: \"kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.063333 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.063211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.164531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.164492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.164531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.164542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n79p\" (UniqueName: \"kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.164825 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.164662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.164825 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.164740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.164825 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.164785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.165052 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.165032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.165334 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.165312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.165421 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.165372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.167396 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.167378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.172593 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.172570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n79p\" (UniqueName: \"kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p\") pod \"isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.240349 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.240318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:13.367822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.367730    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f120bd4-9753-4fb9-9273-d7655aee0df8" containerID="8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220" exitCode=0
Sep 09 20:40:13.367822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.367796    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"
Sep 09 20:40:13.367822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.367808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerDied","Data":"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"}
Sep 09 20:40:13.368039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.367875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw" event={"ID":"9f120bd4-9753-4fb9-9273-d7655aee0df8","Type":"ContainerDied","Data":"cc91f2a0518205676d3218dc59521f83303363678ec9eabb11583d092c7dd733"}
Sep 09 20:40:13.368039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.367918    2579 scope.go:117] "RemoveContainer" containerID="6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"
Sep 09 20:40:13.377614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.377587    2579 scope.go:117] "RemoveContainer" containerID="8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"
Sep 09 20:40:13.386061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.386038    2579 scope.go:117] "RemoveContainer" containerID="fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"
Sep 09 20:40:13.390492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.390464    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:40:13.395007 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.394986    2579 scope.go:117] "RemoveContainer" containerID="6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"
Sep 09 20:40:13.395421 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:13.395399    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6\": container with ID starting with 6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6 not found: ID does not exist" containerID="6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"
Sep 09 20:40:13.395485 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.395430    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6"} err="failed to get container status \"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6\": rpc error: code = NotFound desc = could not find container \"6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6\": container with ID starting with 6faac9d4c9def88f6fa6ce9aeab00db8a9d95c91d3f9fde4afe69832773bb5b6 not found: ID does not exist"
Sep 09 20:40:13.395485 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.395449    2579 scope.go:117] "RemoveContainer" containerID="8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"
Sep 09 20:40:13.395762 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:13.395745    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220\": container with ID starting with 8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220 not found: ID does not exist" containerID="8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"
Sep 09 20:40:13.395812 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.395768    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220"} err="failed to get container status \"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220\": rpc error: code = NotFound desc = could not find container \"8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220\": container with ID starting with 8493ac8c2e7709eec10637a9b3eb02056c02e437636ce6bf6084774600cc0220 not found: ID does not exist"
Sep 09 20:40:13.395812 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.395789    2579 scope.go:117] "RemoveContainer" containerID="fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"
Sep 09 20:40:13.395997 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.395977    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b94c8df76-96hrw"]
Sep 09 20:40:13.396062 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:13.396038    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67\": container with ID starting with fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67 not found: ID does not exist" containerID="fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"
Sep 09 20:40:13.396062 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.396054    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67"} err="failed to get container status \"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67\": rpc error: code = NotFound desc = could not find container \"fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67\": container with ID starting with fb36fa5a43a60fd16678de7235a2e722d4ce7f08aab7448183cbb7a16b3f4e67 not found: ID does not exist"
Sep 09 20:40:13.512264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.512235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:40:13.514593 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:40:13.514559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696f0554_1264_43a5_bc08_89b087b32d04.slice/crio-530747fa65be9e8873c2e0a35e9f8bf4f4659e85c2f59433410096a59f0c84b8 WatchSource:0}: Error finding container 530747fa65be9e8873c2e0a35e9f8bf4f4659e85c2f59433410096a59f0c84b8: Status 404 returned error can't find the container with id 530747fa65be9e8873c2e0a35e9f8bf4f4659e85c2f59433410096a59f0c84b8
Sep 09 20:40:13.611502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.611467    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f120bd4-9753-4fb9-9273-d7655aee0df8" path="/var/lib/kubelet/pods/9f120bd4-9753-4fb9-9273-d7655aee0df8/volumes"
Sep 09 20:40:13.611928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:13.611910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:40:14.374648 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:14.374601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerStarted","Data":"e04d83edf9cd58bae51c83b51eaa78ca922735f7fc7047095c631401ad713fbb"}
Sep 09 20:40:14.375168 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:14.374682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerStarted","Data":"530747fa65be9e8873c2e0a35e9f8bf4f4659e85c2f59433410096a59f0c84b8"}
Sep 09 20:40:18.392341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:18.392303    2579 generic.go:356] "Generic (PLEG): container finished" podID="696f0554-1264-43a5-bc08-89b087b32d04" containerID="e04d83edf9cd58bae51c83b51eaa78ca922735f7fc7047095c631401ad713fbb" exitCode=0
Sep 09 20:40:18.392856 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:18.392389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerDied","Data":"e04d83edf9cd58bae51c83b51eaa78ca922735f7fc7047095c631401ad713fbb"}
Sep 09 20:40:18.605838 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:18.605795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:18.606522 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:18.606495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:19.398459 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.398422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerStarted","Data":"0f4fffa5920e6abb7e0d012e520cfc20a4803b2b774db0f3806a69ae74a0f440"}
Sep 09 20:40:19.398875 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.398487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerStarted","Data":"e298cf92f152e40977f14c7de336d90d6c849ed03c3be1fbe3ce558308ab0f89"}
Sep 09 20:40:19.398875 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.398754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:19.398875 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.398801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:19.400305 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.400278    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:19.416839 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:19.416786    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podStartSLOduration=3.313591431 podStartE2EDuration="7.416768083s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.103176652s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:19.415313561 +0000 UTC m=+1752.251628023" watchObservedRunningTime="2026-09-09 20:40:19.416768083 +0000 UTC m=+1752.253082546"
Sep 09 20:40:20.403283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:20.403241    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:24.647373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:24.647309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:40:25.408282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:25.408245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:40:25.408844 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:25.408815    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:28.490246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:28.490208    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:40:28.490638 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:28.490512    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kserve-container" containerID="cri-o://e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411" gracePeriod=30
Sep 09 20:40:28.490638 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:28.490547    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kube-rbac-proxy" containerID="cri-o://a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04" gracePeriod=30
Sep 09 20:40:28.605914 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:28.605866    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:28.606431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:28.606403    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:29.447351 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:29.447316    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b33e023-bd06-4aee-b74c-4b83611083df" containerID="a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04" exitCode=2
Sep 09 20:40:29.447546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:29.447389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerDied","Data":"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"}
Sep 09 20:40:30.042439 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.042399    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:40:30.214868 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.214828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location\") pod \"5b33e023-bd06-4aee-b74c-4b83611083df\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") "
Sep 09 20:40:30.215057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.214900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls\") pod \"5b33e023-bd06-4aee-b74c-4b83611083df\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") "
Sep 09 20:40:30.215057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.214948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert\") pod \"5b33e023-bd06-4aee-b74c-4b83611083df\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") "
Sep 09 20:40:30.215057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.214990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config\") pod \"5b33e023-bd06-4aee-b74c-4b83611083df\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") "
Sep 09 20:40:30.215057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.215035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw2jr\" (UniqueName: \"kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr\") pod \"5b33e023-bd06-4aee-b74c-4b83611083df\" (UID: \"5b33e023-bd06-4aee-b74c-4b83611083df\") "
Sep 09 20:40:30.215452 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.215418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b33e023-bd06-4aee-b74c-4b83611083df" (UID: "5b33e023-bd06-4aee-b74c-4b83611083df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:30.215557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.215420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b33e023-bd06-4aee-b74c-4b83611083df" (UID: "5b33e023-bd06-4aee-b74c-4b83611083df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:30.217336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.217307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b33e023-bd06-4aee-b74c-4b83611083df" (UID: "5b33e023-bd06-4aee-b74c-4b83611083df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:30.217437 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.217380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr" (OuterVolumeSpecName: "kube-api-access-jw2jr") pod "5b33e023-bd06-4aee-b74c-4b83611083df" (UID: "5b33e023-bd06-4aee-b74c-4b83611083df"). InnerVolumeSpecName "kube-api-access-jw2jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:30.219454 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.219432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b33e023-bd06-4aee-b74c-4b83611083df" (UID: "5b33e023-bd06-4aee-b74c-4b83611083df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:30.315811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.315779    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b33e023-bd06-4aee-b74c-4b83611083df-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:30.315811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.315810    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:30.315811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.315820    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b33e023-bd06-4aee-b74c-4b83611083df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:30.316077 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.315830    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2jr\" (UniqueName: \"kubernetes.io/projected/5b33e023-bd06-4aee-b74c-4b83611083df-kube-api-access-jw2jr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:30.316077 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.315839    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b33e023-bd06-4aee-b74c-4b83611083df-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:40:30.453495 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.453458    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b33e023-bd06-4aee-b74c-4b83611083df" containerID="e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411" exitCode=0
Sep 09 20:40:30.453663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.453526    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"
Sep 09 20:40:30.453663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.453533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerDied","Data":"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"}
Sep 09 20:40:30.453663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.453590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb" event={"ID":"5b33e023-bd06-4aee-b74c-4b83611083df","Type":"ContainerDied","Data":"ab57222df71040f6d199cd26a90bb8220f009c2e6f5d88288a91a47ce36acf23"}
Sep 09 20:40:30.453663 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.453617    2579 scope.go:117] "RemoveContainer" containerID="a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"
Sep 09 20:40:30.462730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.462711    2579 scope.go:117] "RemoveContainer" containerID="e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"
Sep 09 20:40:30.470708 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.470689    2579 scope.go:117] "RemoveContainer" containerID="9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"
Sep 09 20:40:30.476846 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.476811    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:40:30.479614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.479561    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-twxfb"]
Sep 09 20:40:30.481064 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.481048    2579 scope.go:117] "RemoveContainer" containerID="a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"
Sep 09 20:40:30.481463 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:30.481443    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04\": container with ID starting with a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04 not found: ID does not exist" containerID="a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"
Sep 09 20:40:30.481519 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.481472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04"} err="failed to get container status \"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04\": rpc error: code = NotFound desc = could not find container \"a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04\": container with ID starting with a209f788935ae16c9f1fa6847b6306ca6b6a59b147dfcdcca66756b94acdda04 not found: ID does not exist"
Sep 09 20:40:30.481519 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.481493    2579 scope.go:117] "RemoveContainer" containerID="e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"
Sep 09 20:40:30.481750 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:30.481733    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411\": container with ID starting with e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411 not found: ID does not exist" containerID="e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"
Sep 09 20:40:30.481799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.481754    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411"} err="failed to get container status \"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411\": rpc error: code = NotFound desc = could not find container \"e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411\": container with ID starting with e0cc12259ba220e053b09ee915115a2e0af45c03e267be6fc89dbccbdba20411 not found: ID does not exist"
Sep 09 20:40:30.481799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.481772    2579 scope.go:117] "RemoveContainer" containerID="9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"
Sep 09 20:40:30.482062 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:40:30.482045    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc\": container with ID starting with 9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc not found: ID does not exist" containerID="9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"
Sep 09 20:40:30.482142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:30.482072    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc"} err="failed to get container status \"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc\": rpc error: code = NotFound desc = could not find container \"9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc\": container with ID starting with 9bae5db16f34e47741bd2044c5f6566cb65f91d4757419c86526b0eb2a10cdbc not found: ID does not exist"
Sep 09 20:40:31.610774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:31.610737    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" path="/var/lib/kubelet/pods/5b33e023-bd06-4aee-b74c-4b83611083df/volumes"
Sep 09 20:40:33.660881 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.660836    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661342    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="storage-initializer"
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661363    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="storage-initializer"
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661392    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kube-rbac-proxy"
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661401    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kube-rbac-proxy"
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661413    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kserve-container"
Sep 09 20:40:33.661420 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661422    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kserve-container"
Sep 09 20:40:33.661763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661503    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kserve-container"
Sep 09 20:40:33.661763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.661517    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b33e023-bd06-4aee-b74c-4b83611083df" containerName="kube-rbac-proxy"
Sep 09 20:40:33.666780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.666754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.669533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.669506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 09 20:40:33.669533 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.669524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 09 20:40:33.673531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.673500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:40:33.742878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.742834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2v2s\" (UniqueName: \"kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.743056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.742892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.743056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.742951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.743056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.743002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.743056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.743041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2v2s\" (UniqueName: \"kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844756 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.844870 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.844850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.845197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.845177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.845285 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.845191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.847039 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.847018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.852554 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.852527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2v2s\" (UniqueName: \"kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:33.978837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:33.978743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:34.253350 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:34.253313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:40:34.255652 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:40:34.255620    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cf8bb9_5489_4e9e_9cdc_644203699740.slice/crio-5d732e3b2bfa22a3c6d30423e73157ef60c462c6ba3725b2329b8ed6b4bf7a9b WatchSource:0}: Error finding container 5d732e3b2bfa22a3c6d30423e73157ef60c462c6ba3725b2329b8ed6b4bf7a9b: Status 404 returned error can't find the container with id 5d732e3b2bfa22a3c6d30423e73157ef60c462c6ba3725b2329b8ed6b4bf7a9b
Sep 09 20:40:34.473393 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:34.473354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerStarted","Data":"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"}
Sep 09 20:40:34.473549 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:34.473422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerStarted","Data":"5d732e3b2bfa22a3c6d30423e73157ef60c462c6ba3725b2329b8ed6b4bf7a9b"}
Sep 09 20:40:35.409735 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:35.409693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:38.605873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:38.605837    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:38.606343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:38.606320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:40.501516 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:40.501478    2579 generic.go:356] "Generic (PLEG): container finished" podID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerID="4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83" exitCode=0
Sep 09 20:40:40.501911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:40.501550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerDied","Data":"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"}
Sep 09 20:40:41.508415 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:41.508380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerStarted","Data":"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"}
Sep 09 20:40:41.508835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:41.508446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerStarted","Data":"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"}
Sep 09 20:40:41.508835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:41.508669    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:41.508918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:41.508839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:41.526944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:41.526883    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" podStartSLOduration=3.293997073 podStartE2EDuration="8.526865539s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.232868466s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.526675548 +0000 UTC m=+1774.362990024" watchObservedRunningTime="2026-09-09 20:40:41.526865539 +0000 UTC m=+1774.363179984"
Sep 09 20:40:45.408877 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:45.408836    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:47.518963 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:47.518929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:40:48.606418 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:48.606362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:48.606911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:48.606776    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:40:55.409242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:55.409199    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:40:58.606256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:58.606153    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:40:58.606639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:40:58.606549    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:05.408737 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:05.408695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:41:08.605891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:08.605842    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:08.606403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:08.606281    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:15.409260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:15.409210    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:41:18.606428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:18.606378    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:18.606877 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:18.606754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:25.408855 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:25.408814    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:41:28.606382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:28.606327    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:28.606850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:28.606820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:35.408883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:35.408836    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:41:35.606362 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:35.606311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:35.606657 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:35.606636    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:45.409881 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:45.409843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:41:45.606944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:45.606892    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:45.607399 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:45.607370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:52.981695 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:52.981654    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:41:52.982301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:52.981961    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" containerID="cri-o://e298cf92f152e40977f14c7de336d90d6c849ed03c3be1fbe3ce558308ab0f89" gracePeriod=30
Sep 09 20:41:52.982301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:52.981994    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kube-rbac-proxy" containerID="cri-o://0f4fffa5920e6abb7e0d012e520cfc20a4803b2b774db0f3806a69ae74a0f440" gracePeriod=30
Sep 09 20:41:53.801023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:53.800991    2579 generic.go:356] "Generic (PLEG): container finished" podID="696f0554-1264-43a5-bc08-89b087b32d04" containerID="0f4fffa5920e6abb7e0d012e520cfc20a4803b2b774db0f3806a69ae74a0f440" exitCode=2
Sep 09 20:41:53.801217 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:53.801051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerDied","Data":"0f4fffa5920e6abb7e0d012e520cfc20a4803b2b774db0f3806a69ae74a0f440"}
Sep 09 20:41:55.403627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:55.403580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 09 20:41:55.408737 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:55.408706    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 09 20:41:55.606292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:55.606237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:41:55.606704 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:55.606678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:41:57.820546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:57.820511    2579 generic.go:356] "Generic (PLEG): container finished" podID="696f0554-1264-43a5-bc08-89b087b32d04" containerID="e298cf92f152e40977f14c7de336d90d6c849ed03c3be1fbe3ce558308ab0f89" exitCode=0
Sep 09 20:41:57.820935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:57.820584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerDied","Data":"e298cf92f152e40977f14c7de336d90d6c849ed03c3be1fbe3ce558308ab0f89"}
Sep 09 20:41:57.936787 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:57.936757    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:41:58.041112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls\") pod \"696f0554-1264-43a5-bc08-89b087b32d04\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") "
Sep 09 20:41:58.041112 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert\") pod \"696f0554-1264-43a5-bc08-89b087b32d04\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") "
Sep 09 20:41:58.041353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location\") pod \"696f0554-1264-43a5-bc08-89b087b32d04\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") "
Sep 09 20:41:58.041353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config\") pod \"696f0554-1264-43a5-bc08-89b087b32d04\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") "
Sep 09 20:41:58.041353 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n79p\" (UniqueName: \"kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p\") pod \"696f0554-1264-43a5-bc08-89b087b32d04\" (UID: \"696f0554-1264-43a5-bc08-89b087b32d04\") "
Sep 09 20:41:58.041517 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "696f0554-1264-43a5-bc08-89b087b32d04" (UID: "696f0554-1264-43a5-bc08-89b087b32d04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:58.041517 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "696f0554-1264-43a5-bc08-89b087b32d04" (UID: "696f0554-1264-43a5-bc08-89b087b32d04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:58.041630 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.041577    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "696f0554-1264-43a5-bc08-89b087b32d04" (UID: "696f0554-1264-43a5-bc08-89b087b32d04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:58.043443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.043418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "696f0554-1264-43a5-bc08-89b087b32d04" (UID: "696f0554-1264-43a5-bc08-89b087b32d04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:58.043443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.043431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p" (OuterVolumeSpecName: "kube-api-access-8n79p") pod "696f0554-1264-43a5-bc08-89b087b32d04" (UID: "696f0554-1264-43a5-bc08-89b087b32d04"). InnerVolumeSpecName "kube-api-access-8n79p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:58.142275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.142237    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:41:58.142275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.142270    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/696f0554-1264-43a5-bc08-89b087b32d04-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:41:58.142493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.142285    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/696f0554-1264-43a5-bc08-89b087b32d04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:41:58.142493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.142297    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n79p\" (UniqueName: \"kubernetes.io/projected/696f0554-1264-43a5-bc08-89b087b32d04-kube-api-access-8n79p\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:41:58.142493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.142311    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/696f0554-1264-43a5-bc08-89b087b32d04-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:41:58.827222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.827195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"
Sep 09 20:41:58.827222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.827206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz" event={"ID":"696f0554-1264-43a5-bc08-89b087b32d04","Type":"ContainerDied","Data":"530747fa65be9e8873c2e0a35e9f8bf4f4659e85c2f59433410096a59f0c84b8"}
Sep 09 20:41:58.827744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.827293    2579 scope.go:117] "RemoveContainer" containerID="0f4fffa5920e6abb7e0d012e520cfc20a4803b2b774db0f3806a69ae74a0f440"
Sep 09 20:41:58.836586 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.836570    2579 scope.go:117] "RemoveContainer" containerID="e298cf92f152e40977f14c7de336d90d6c849ed03c3be1fbe3ce558308ab0f89"
Sep 09 20:41:58.844162 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.844146    2579 scope.go:117] "RemoveContainer" containerID="e04d83edf9cd58bae51c83b51eaa78ca922735f7fc7047095c631401ad713fbb"
Sep 09 20:41:58.849070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.849044    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:41:58.852422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:58.852399    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6495c49d49-c42nz"]
Sep 09 20:41:59.611190 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:41:59.611158    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696f0554-1264-43a5-bc08-89b087b32d04" path="/var/lib/kubelet/pods/696f0554-1264-43a5-bc08-89b087b32d04/volumes"
Sep 09 20:42:05.606348 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:05.606301    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:05.608921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:05.606755    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:12.610142 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:12.610105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:42:15.606360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:15.606294    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:15.606867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:15.606797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:20.786636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:20.786598    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:42:20.787049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:20.786864    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kserve-container" containerID="cri-o://29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055" gracePeriod=30
Sep 09 20:42:20.787049 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:20.786887    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kube-rbac-proxy" containerID="cri-o://d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158" gracePeriod=30
Sep 09 20:42:20.922711 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:20.922679    2579 generic.go:356] "Generic (PLEG): container finished" podID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerID="d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158" exitCode=2
Sep 09 20:42:20.922856 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:20.922749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerDied","Data":"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"}
Sep 09 20:42:22.322435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.322409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:42:22.443761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.443714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls\") pod \"54cf8bb9-5489-4e9e-9cdc-644203699740\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") "
Sep 09 20:42:22.443958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.443776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2v2s\" (UniqueName: \"kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s\") pod \"54cf8bb9-5489-4e9e-9cdc-644203699740\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") "
Sep 09 20:42:22.443958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.443892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert\") pod \"54cf8bb9-5489-4e9e-9cdc-644203699740\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") "
Sep 09 20:42:22.443958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.443924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config\") pod \"54cf8bb9-5489-4e9e-9cdc-644203699740\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") "
Sep 09 20:42:22.444075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.443978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location\") pod \"54cf8bb9-5489-4e9e-9cdc-644203699740\" (UID: \"54cf8bb9-5489-4e9e-9cdc-644203699740\") "
Sep 09 20:42:22.444338 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.444306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54cf8bb9-5489-4e9e-9cdc-644203699740" (UID: "54cf8bb9-5489-4e9e-9cdc-644203699740"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:42:22.444338 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.444317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54cf8bb9-5489-4e9e-9cdc-644203699740" (UID: "54cf8bb9-5489-4e9e-9cdc-644203699740"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:42:22.446269 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.446236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54cf8bb9-5489-4e9e-9cdc-644203699740" (UID: "54cf8bb9-5489-4e9e-9cdc-644203699740"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:22.446269 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.446247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s" (OuterVolumeSpecName: "kube-api-access-l2v2s") pod "54cf8bb9-5489-4e9e-9cdc-644203699740" (UID: "54cf8bb9-5489-4e9e-9cdc-644203699740"). InnerVolumeSpecName "kube-api-access-l2v2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:22.448988 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.448965    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54cf8bb9-5489-4e9e-9cdc-644203699740" (UID: "54cf8bb9-5489-4e9e-9cdc-644203699740"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:22.545435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.545389    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.545435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.545432    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.545435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.545442    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54cf8bb9-5489-4e9e-9cdc-644203699740-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.545435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.545452    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cf8bb9-5489-4e9e-9cdc-644203699740-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.545718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.545463    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2v2s\" (UniqueName: \"kubernetes.io/projected/54cf8bb9-5489-4e9e-9cdc-644203699740-kube-api-access-l2v2s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:42:22.934136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.934079    2579 generic.go:356] "Generic (PLEG): container finished" podID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerID="29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055" exitCode=0
Sep 09 20:42:22.934307 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.934151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerDied","Data":"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"}
Sep 09 20:42:22.934307 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.934168    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"
Sep 09 20:42:22.934307 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.934208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7" event={"ID":"54cf8bb9-5489-4e9e-9cdc-644203699740","Type":"ContainerDied","Data":"5d732e3b2bfa22a3c6d30423e73157ef60c462c6ba3725b2329b8ed6b4bf7a9b"}
Sep 09 20:42:22.934307 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.934236    2579 scope.go:117] "RemoveContainer" containerID="d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"
Sep 09 20:42:22.942677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.942657    2579 scope.go:117] "RemoveContainer" containerID="29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"
Sep 09 20:42:22.950859 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.950840    2579 scope.go:117] "RemoveContainer" containerID="4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"
Sep 09 20:42:22.955795 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.955761    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:42:22.958202 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.958179    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-f5hv7"]
Sep 09 20:42:22.960213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.960191    2579 scope.go:117] "RemoveContainer" containerID="d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"
Sep 09 20:42:22.960551 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:42:22.960531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158\": container with ID starting with d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158 not found: ID does not exist" containerID="d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"
Sep 09 20:42:22.960632 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.960565    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158"} err="failed to get container status \"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158\": rpc error: code = NotFound desc = could not find container \"d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158\": container with ID starting with d5072dbb660eead1a449fbf03d6bf89463796647c9106f3c28af9fb10e4e6158 not found: ID does not exist"
Sep 09 20:42:22.960632 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.960591    2579 scope.go:117] "RemoveContainer" containerID="29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"
Sep 09 20:42:22.960896 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:42:22.960878    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055\": container with ID starting with 29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055 not found: ID does not exist" containerID="29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"
Sep 09 20:42:22.960967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.960904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055"} err="failed to get container status \"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055\": rpc error: code = NotFound desc = could not find container \"29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055\": container with ID starting with 29c67b458fa74230ed4f113773d175246a0116fcfb637d59e3d5b3a59a972055 not found: ID does not exist"
Sep 09 20:42:22.960967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.960927    2579 scope.go:117] "RemoveContainer" containerID="4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"
Sep 09 20:42:22.961214 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:42:22.961198    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83\": container with ID starting with 4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83 not found: ID does not exist" containerID="4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"
Sep 09 20:42:22.961278 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:22.961221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83"} err="failed to get container status \"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83\": rpc error: code = NotFound desc = could not find container \"4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83\": container with ID starting with 4a7429d5d921f0ce5a233a2081d65a713f561e72684e595146b19bc65cb40d83 not found: ID does not exist"
Sep 09 20:42:23.610680 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:23.610646    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" path="/var/lib/kubelet/pods/54cf8bb9-5489-4e9e-9cdc-644203699740/volumes"
Sep 09 20:42:25.606247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:25.606197    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:25.606730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:25.606587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:26.059226 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059189    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:42:26.059525 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059509    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kserve-container"
Sep 09 20:42:26.059579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kserve-container"
Sep 09 20:42:26.059579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059547    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059556    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="storage-initializer"
Sep 09 20:42:26.059579 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059570    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="storage-initializer"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059583    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059590    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059603    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059610    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059618    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="storage-initializer"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="storage-initializer"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059674    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kserve-container"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059685    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf8bb9-5489-4e9e-9cdc-644203699740" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059693    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kube-rbac-proxy"
Sep 09 20:42:26.059744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.059704    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="696f0554-1264-43a5-bc08-89b087b32d04" containerName="kserve-container"
Sep 09 20:42:26.064384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.064364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.066734 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.066705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 09 20:42:26.066858 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.066812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 09 20:42:26.070996 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.070970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:42:26.173224 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.173189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67fg\" (UniqueName: \"kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.173403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.173251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.173403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.173316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.173403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.173359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.173403 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.173387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274372 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67fg\" (UniqueName: \"kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.274884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.274856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.275137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.275083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.275216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.275184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.277356 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.277330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.282172 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.282141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67fg\" (UniqueName: \"kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.375627 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.375526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:26.651068 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.651042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:42:26.653351 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:42:26.653323    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda763d7b4_e039_4c56_bab6_878a6d880579.slice/crio-9427db90899586ab11392a8724435f08817a7a548b3933ee48db682abdfc37bd WatchSource:0}: Error finding container 9427db90899586ab11392a8724435f08817a7a548b3933ee48db682abdfc37bd: Status 404 returned error can't find the container with id 9427db90899586ab11392a8724435f08817a7a548b3933ee48db682abdfc37bd
Sep 09 20:42:26.959862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.959814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerStarted","Data":"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"}
Sep 09 20:42:26.960080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:26.959928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerStarted","Data":"9427db90899586ab11392a8724435f08817a7a548b3933ee48db682abdfc37bd"}
Sep 09 20:42:35.606407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:35.606352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:35.606876 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:35.606854    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:45.606909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:45.606861    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:45.607373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:45.607349    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:48.048050 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:48.048009    2579 generic.go:356] "Generic (PLEG): container finished" podID="a763d7b4-e039-4c56-bab6-878a6d880579" containerID="3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6" exitCode=0
Sep 09 20:42:48.048494 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:48.048074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerDied","Data":"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"}
Sep 09 20:42:49.054233 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:49.054193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerStarted","Data":"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"}
Sep 09 20:42:49.054637 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:49.054286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerStarted","Data":"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"}
Sep 09 20:42:49.054637 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:49.054604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:49.054729 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:49.054665    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:49.074149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:49.074066    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" podStartSLOduration=2.225042109 podStartE2EDuration="23.074051267s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.849009158s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:49.072547774 +0000 UTC m=+1901.908862261" watchObservedRunningTime="2026-09-09 20:42:49.074051267 +0000 UTC m=+1901.910365730"
Sep 09 20:42:55.064110 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:55.064058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:42:55.606967 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:55.606915    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:55.607500 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:55.607457    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:42:57.609380 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:57.609341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:42:57.609932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:42:57.609797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:07.609385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:07.609332    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:07.610037 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:07.609783    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:17.609822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:17.609782    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:17.610322 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:17.610200    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:27.609907 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:27.609861    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:27.610453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:27.610425    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:37.610824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:37.610765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:37.612747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:37.611667    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:47.609316 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:47.609273    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:47.609936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:47.609741    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:43:57.609314 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:57.609280    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:43:57.609777 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:43:57.609757    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:07.609676 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:07.609478    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:07.610120 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:07.610010    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:14.606236 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:14.606187    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:14.606677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:14.606645    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:21.615561 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:21.615524    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:44:24.607078 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:24.607015    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:24.607512 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:24.607482    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:32.616626 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:32.616592    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:44:32.617056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:32.616838    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kserve-container" containerID="cri-o://3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6" gracePeriod=30
Sep 09 20:44:32.617056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:32.616943    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kube-rbac-proxy" containerID="cri-o://828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9" gracePeriod=30
Sep 09 20:44:33.502020 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:33.501979    2579 generic.go:356] "Generic (PLEG): container finished" podID="a763d7b4-e039-4c56-bab6-878a6d880579" containerID="828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9" exitCode=2
Sep 09 20:44:33.502253 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:33.502017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerDied","Data":"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"}
Sep 09 20:44:34.178578 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.178549    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:44:34.267457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config\") pod \"a763d7b4-e039-4c56-bab6-878a6d880579\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") "
Sep 09 20:44:34.267643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j67fg\" (UniqueName: \"kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg\") pod \"a763d7b4-e039-4c56-bab6-878a6d880579\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") "
Sep 09 20:44:34.267643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location\") pod \"a763d7b4-e039-4c56-bab6-878a6d880579\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") "
Sep 09 20:44:34.267643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls\") pod \"a763d7b4-e039-4c56-bab6-878a6d880579\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") "
Sep 09 20:44:34.267643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267591    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert\") pod \"a763d7b4-e039-4c56-bab6-878a6d880579\" (UID: \"a763d7b4-e039-4c56-bab6-878a6d880579\") "
Sep 09 20:44:34.268017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.267889    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a763d7b4-e039-4c56-bab6-878a6d880579" (UID: "a763d7b4-e039-4c56-bab6-878a6d880579"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:34.268073 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.268042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a763d7b4-e039-4c56-bab6-878a6d880579" (UID: "a763d7b4-e039-4c56-bab6-878a6d880579"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:34.269920 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.269866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg" (OuterVolumeSpecName: "kube-api-access-j67fg") pod "a763d7b4-e039-4c56-bab6-878a6d880579" (UID: "a763d7b4-e039-4c56-bab6-878a6d880579"). InnerVolumeSpecName "kube-api-access-j67fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:44:34.269989 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.269940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a763d7b4-e039-4c56-bab6-878a6d880579" (UID: "a763d7b4-e039-4c56-bab6-878a6d880579"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:44:34.272300 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.272274    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a763d7b4-e039-4c56-bab6-878a6d880579" (UID: "a763d7b4-e039-4c56-bab6-878a6d880579"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:44:34.368195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.368155    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:44:34.368195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.368189    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67fg\" (UniqueName: \"kubernetes.io/projected/a763d7b4-e039-4c56-bab6-878a6d880579-kube-api-access-j67fg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:44:34.368195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.368202    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a763d7b4-e039-4c56-bab6-878a6d880579-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:44:34.368441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.368214    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a763d7b4-e039-4c56-bab6-878a6d880579-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:44:34.368441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.368228    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a763d7b4-e039-4c56-bab6-878a6d880579-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:44:34.508644 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.508608    2579 generic.go:356] "Generic (PLEG): container finished" podID="a763d7b4-e039-4c56-bab6-878a6d880579" containerID="3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6" exitCode=0
Sep 09 20:44:34.508817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.508671    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"
Sep 09 20:44:34.508817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.508695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerDied","Data":"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"}
Sep 09 20:44:34.508817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.508799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x" event={"ID":"a763d7b4-e039-4c56-bab6-878a6d880579","Type":"ContainerDied","Data":"9427db90899586ab11392a8724435f08817a7a548b3933ee48db682abdfc37bd"}
Sep 09 20:44:34.508933 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.508840    2579 scope.go:117] "RemoveContainer" containerID="828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"
Sep 09 20:44:34.517721 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.517700    2579 scope.go:117] "RemoveContainer" containerID="3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"
Sep 09 20:44:34.525581 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.525553    2579 scope.go:117] "RemoveContainer" containerID="3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"
Sep 09 20:44:34.529584 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.529553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:44:34.532875 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.532845    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-fjg5x"]
Sep 09 20:44:34.534521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.534502    2579 scope.go:117] "RemoveContainer" containerID="828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"
Sep 09 20:44:34.534862 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:44:34.534838    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9\": container with ID starting with 828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9 not found: ID does not exist" containerID="828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"
Sep 09 20:44:34.534908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.534873    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9"} err="failed to get container status \"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9\": rpc error: code = NotFound desc = could not find container \"828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9\": container with ID starting with 828d2214602b1d94b71dd80f2729f289286cb03c2a14ce22b1d74c48b5b787e9 not found: ID does not exist"
Sep 09 20:44:34.534908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.534893    2579 scope.go:117] "RemoveContainer" containerID="3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"
Sep 09 20:44:34.535207 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:44:34.535189    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6\": container with ID starting with 3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6 not found: ID does not exist" containerID="3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"
Sep 09 20:44:34.535272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.535213    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6"} err="failed to get container status \"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6\": rpc error: code = NotFound desc = could not find container \"3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6\": container with ID starting with 3ef7de2c5daa8ab6d3d55eb288487cb69c32504c98bb2157fa8f69dd3784fea6 not found: ID does not exist"
Sep 09 20:44:34.535272 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.535230    2579 scope.go:117] "RemoveContainer" containerID="3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"
Sep 09 20:44:34.535534 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:44:34.535513    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6\": container with ID starting with 3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6 not found: ID does not exist" containerID="3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"
Sep 09 20:44:34.535603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.535551    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6"} err="failed to get container status \"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6\": rpc error: code = NotFound desc = could not find container \"3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6\": container with ID starting with 3c42a98715e51287ea0239f842c8eab8a415100df620933fcce8f9329416b6a6 not found: ID does not exist"
Sep 09 20:44:34.606899 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.606854    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:34.607338 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:34.607313    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:35.611124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:35.611070    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" path="/var/lib/kubelet/pods/a763d7b4-e039-4c56-bab6-878a6d880579/volumes"
Sep 09 20:44:44.606468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:44.606419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:44.606875 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:44.606853    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:44:54.606340 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:54.606236    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:44:54.606884 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:44:54.606748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:04.606647 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:04.606592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:04.607140 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:04.607115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:14.607048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:14.606993    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:14.607487 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:14.607449    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:24.606262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:24.606212    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:24.606710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:24.606680    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:34.606910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:34.606864    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:34.607411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:34.607390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:36.606629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:36.606585    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:36.607032 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:36.606916    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:46.607508 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:46.607453    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:46.607958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:46.607932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:45:56.606562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:56.606511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:45:56.607008 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:45:56.606938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:06.607431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:06.607384    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:06.608109 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:06.607835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:09.171058 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171019    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171521    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kserve-container"
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kserve-container"
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171586    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="storage-initializer"
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171595    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="storage-initializer"
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171608    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kube-rbac-proxy"
Sep 09 20:46:09.171631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kube-rbac-proxy"
Sep 09 20:46:09.171987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171702    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kserve-container"
Sep 09 20:46:09.171987 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.171717    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a763d7b4-e039-4c56-bab6-878a6d880579" containerName="kube-rbac-proxy"
Sep 09 20:46:09.175156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.175135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.177928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.177897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 09 20:46:09.177928 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.177925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 20:46:09.184720 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.184700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:46:09.276398 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.276362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.276398 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.276404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gx8\" (UniqueName: \"kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.276671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.276435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.276671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.276518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.276671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.276582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.377739 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.377697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.377915 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.377795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gx8\" (UniqueName: \"kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.377915 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.377873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.378034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.377922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.378034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.377988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.378034 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:46:09.378014    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 09 20:46:09.378210 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:46:09.378100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls podName:1e5f5853-73fb-42d4-b636-bba255f560e7 nodeName:}" failed. No retries permitted until 2026-09-09 20:46:09.878062292 +0000 UTC m=+2102.714376733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-l4m99" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 09 20:46:09.378259 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.378236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.378830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.378620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.379206 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.379183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.386972 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.386939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gx8\" (UniqueName: \"kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.882870 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.882828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:09.885603 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:09.885574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-l4m99\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:10.085770 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:10.085727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:10.356545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:10.356518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:46:10.358728 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:46:10.358698    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5f5853_73fb_42d4_b636_bba255f560e7.slice/crio-ae71275e3de09564512d718696b7838da2e06000965d38495990110c526d9972 WatchSource:0}: Error finding container ae71275e3de09564512d718696b7838da2e06000965d38495990110c526d9972: Status 404 returned error can't find the container with id ae71275e3de09564512d718696b7838da2e06000965d38495990110c526d9972
Sep 09 20:46:10.898854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:10.898810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerStarted","Data":"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"}
Sep 09 20:46:10.899031 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:10.898893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerStarted","Data":"ae71275e3de09564512d718696b7838da2e06000965d38495990110c526d9972"}
Sep 09 20:46:14.915874 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:14.915839    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerID="b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d" exitCode=0
Sep 09 20:46:14.916379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:14.915913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerDied","Data":"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"}
Sep 09 20:46:14.917317 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:14.917302    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:46:16.607021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:16.606975    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:16.607473 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:16.607447    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:26.607104 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:26.607049    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:26.607576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:26.607492    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:36.607488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:36.607440    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:36.607940 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:36.607915    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:46.606923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:46.606846    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:46.607519 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:46.607391    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:53.078709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.078677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerStarted","Data":"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"}
Sep 09 20:46:53.079136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.078741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerStarted","Data":"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"}
Sep 09 20:46:53.079136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.079023    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:53.079136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.079079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:53.080523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.080493    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:46:53.097963 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:53.097914    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podStartSLOduration=2.250723491 podStartE2EDuration="44.097899306s" totalImagesPullingTime="37.959834457s" totalInitContainerRuntime="3.887341358s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:53.095902848 +0000 UTC m=+2145.932217322" watchObservedRunningTime="2026-09-09 20:46:53.097899306 +0000 UTC m=+2145.934213769"
Sep 09 20:46:54.083652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:54.083604    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:46:56.606730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:56.606677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:46:56.607205 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:56.607183    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:59.088176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:59.088076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:46:59.088619 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:46:59.088592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:00.606760 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:00.606716    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:00.607209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:00.607118    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:09.089185 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:09.089137    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:10.607512 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:10.607464    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:10.607976 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:10.607951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:19.088958 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:19.088907    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:20.606847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:20.606792    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:20.607411 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:20.607386    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:29.089435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:29.089394    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:30.606614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:30.606574    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:30.607055 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:30.607017    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:39.089364 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:39.089319    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:40.607363 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:40.607319    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:40.607788 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:40.607757    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:49.089310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:49.089267    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 09 20:47:50.606943 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:50.606893    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:47:50.607531 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:50.607324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:47:59.089757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:47:59.089725    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:48:00.606754 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:00.606696    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:00.607176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:00.607144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:09.269900 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:09.269864    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:48:09.270330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:09.270195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container" containerID="cri-o://dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d" gracePeriod=30
Sep 09 20:48:09.270330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:09.270265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kube-rbac-proxy" containerID="cri-o://eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c" gracePeriod=30
Sep 09 20:48:09.395123 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:09.395075    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerID="eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c" exitCode=2
Sep 09 20:48:09.395252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:09.395123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerDied","Data":"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"}
Sep 09 20:48:10.606979 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:10.606936    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:10.607559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:10.607534    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:13.108776 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.108750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:48:13.217602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.217570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert\") pod \"1e5f5853-73fb-42d4-b636-bba255f560e7\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") "
Sep 09 20:48:13.217799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.217626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5gx8\" (UniqueName: \"kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8\") pod \"1e5f5853-73fb-42d4-b636-bba255f560e7\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") "
Sep 09 20:48:13.217799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.217656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") pod \"1e5f5853-73fb-42d4-b636-bba255f560e7\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") "
Sep 09 20:48:13.217799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.217674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config\") pod \"1e5f5853-73fb-42d4-b636-bba255f560e7\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") "
Sep 09 20:48:13.217799 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.217731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location\") pod \"1e5f5853-73fb-42d4-b636-bba255f560e7\" (UID: \"1e5f5853-73fb-42d4-b636-bba255f560e7\") "
Sep 09 20:48:13.218067 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.218037    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e5f5853-73fb-42d4-b636-bba255f560e7" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:13.218179 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.218076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e5f5853-73fb-42d4-b636-bba255f560e7" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:48:13.218179 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.218079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e5f5853-73fb-42d4-b636-bba255f560e7" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:13.219917 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.219896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8" (OuterVolumeSpecName: "kube-api-access-f5gx8") pod "1e5f5853-73fb-42d4-b636-bba255f560e7" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7"). InnerVolumeSpecName "kube-api-access-f5gx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:13.220018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.219995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e5f5853-73fb-42d4-b636-bba255f560e7" (UID: "1e5f5853-73fb-42d4-b636-bba255f560e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:48:13.319030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.318989    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e5f5853-73fb-42d4-b636-bba255f560e7-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:48:13.319030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.319024    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:48:13.319030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.319035    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gx8\" (UniqueName: \"kubernetes.io/projected/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-api-access-f5gx8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:48:13.319301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.319045    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e5f5853-73fb-42d4-b636-bba255f560e7-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:48:13.319301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.319056    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e5f5853-73fb-42d4-b636-bba255f560e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:48:13.413601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.413567    2579 generic.go:356] "Generic (PLEG): container finished" podID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerID="dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d" exitCode=0
Sep 09 20:48:13.413751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.413626    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"
Sep 09 20:48:13.413751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.413647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerDied","Data":"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"}
Sep 09 20:48:13.413751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.413705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99" event={"ID":"1e5f5853-73fb-42d4-b636-bba255f560e7","Type":"ContainerDied","Data":"ae71275e3de09564512d718696b7838da2e06000965d38495990110c526d9972"}
Sep 09 20:48:13.413751 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.413734    2579 scope.go:117] "RemoveContainer" containerID="eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"
Sep 09 20:48:13.423013 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.422996    2579 scope.go:117] "RemoveContainer" containerID="dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"
Sep 09 20:48:13.430749 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.430731    2579 scope.go:117] "RemoveContainer" containerID="b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"
Sep 09 20:48:13.436621 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.436595    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:48:13.439006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.438990    2579 scope.go:117] "RemoveContainer" containerID="eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"
Sep 09 20:48:13.439361 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:48:13.439341    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c\": container with ID starting with eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c not found: ID does not exist" containerID="eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"
Sep 09 20:48:13.439441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.439375    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c"} err="failed to get container status \"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c\": rpc error: code = NotFound desc = could not find container \"eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c\": container with ID starting with eb2c0b379c81c7776bb1038445133837ee1bec970084eebc94c7418fcdbca52c not found: ID does not exist"
Sep 09 20:48:13.439441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.439401    2579 scope.go:117] "RemoveContainer" containerID="dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"
Sep 09 20:48:13.439741 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:48:13.439720    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d\": container with ID starting with dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d not found: ID does not exist" containerID="dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"
Sep 09 20:48:13.439793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.439748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d"} err="failed to get container status \"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d\": rpc error: code = NotFound desc = could not find container \"dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d\": container with ID starting with dac51b6e804ae11efeeb8eb0693e6f6176e6004b6f65beb1e7dde23d0eb20f9d not found: ID does not exist"
Sep 09 20:48:13.439793 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.439765    2579 scope.go:117] "RemoveContainer" containerID="b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"
Sep 09 20:48:13.440052 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:48:13.440034    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d\": container with ID starting with b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d not found: ID does not exist" containerID="b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"
Sep 09 20:48:13.440133 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.440058    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d"} err="failed to get container status \"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d\": rpc error: code = NotFound desc = could not find container \"b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d\": container with ID starting with b1c008d8ae3cd358e73dafdaa9835053d0b38941e56e4bd683cd9128ea04059d not found: ID does not exist"
Sep 09 20:48:13.444276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.444254    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-l4m99"]
Sep 09 20:48:13.611327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:13.611252    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" path="/var/lib/kubelet/pods/1e5f5853-73fb-42d4-b636-bba255f560e7/volumes"
Sep 09 20:48:14.466306 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466274    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466576    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kube-rbac-proxy"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466588    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kube-rbac-proxy"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466598    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="storage-initializer"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466603    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="storage-initializer"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466626    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466675    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kube-rbac-proxy"
Sep 09 20:48:14.466701 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.466683    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5f5853-73fb-42d4-b636-bba255f560e7" containerName="kserve-container"
Sep 09 20:48:14.469867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.469847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.472422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.472397    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 20:48:14.472529 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.472409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:48:14.480197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.480171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:48:14.528866 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.528828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.529041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.528881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.529041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.528924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.529041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.528972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h95k\" (UniqueName: \"kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.529041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.529005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630058 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630236 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630236 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h95k\" (UniqueName: \"kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630236 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630236 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630442 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:48:14.630291    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 09 20:48:14.630442 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:48:14.630387    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls podName:3ba71c99-0005-4e88-8072-92fc2d1aa996 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:15.130365473 +0000 UTC m=+2227.966679932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 09 20:48:14.630543 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630853 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.630934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.630878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:14.638837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:14.638818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h95k\" (UniqueName: \"kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:15.134645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:15.134605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:15.137247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:15.137225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:15.381051 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:15.381015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:15.655898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:15.655871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:48:15.658454 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:48:15.658423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba71c99_0005_4e88_8072_92fc2d1aa996.slice/crio-349a2b142564fd513b1a4dab9e2f1a0e2464cb87330c330f736e942ddc8a28da WatchSource:0}: Error finding container 349a2b142564fd513b1a4dab9e2f1a0e2464cb87330c330f736e942ddc8a28da: Status 404 returned error can't find the container with id 349a2b142564fd513b1a4dab9e2f1a0e2464cb87330c330f736e942ddc8a28da
Sep 09 20:48:16.430188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:16.430148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerStarted","Data":"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"}
Sep 09 20:48:16.430373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:16.430212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerStarted","Data":"349a2b142564fd513b1a4dab9e2f1a0e2464cb87330c330f736e942ddc8a28da"}
Sep 09 20:48:16.606757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:16.606715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:16.607028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:16.607001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:20.450613 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:20.450580    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerID="6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3" exitCode=0
Sep 09 20:48:20.450969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:20.450650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerDied","Data":"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"}
Sep 09 20:48:21.456712 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:21.456671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerStarted","Data":"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"}
Sep 09 20:48:21.457328 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:21.456788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerStarted","Data":"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"}
Sep 09 20:48:21.457328 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:21.457025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:21.478999 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:21.478942    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" podStartSLOduration=3.476914444 podStartE2EDuration="7.478927221s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.002012777s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:21.478190005 +0000 UTC m=+2234.314504469" watchObservedRunningTime="2026-09-09 20:48:21.478927221 +0000 UTC m=+2234.315241687"
Sep 09 20:48:22.461859 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:22.461820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:26.607155 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:26.607081    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:26.607534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:26.607507    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:28.470978 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:28.470946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:48:36.606824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:36.606771    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:36.607330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:36.607240    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:46.606745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:46.606696    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:46.607216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:46.607123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:56.606983 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:56.606929    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:48:56.607493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:56.607416    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:48:58.475361 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:48:58.475330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:49:04.454814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:04.454781    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:49:04.455305 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:04.455044    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kserve-container" containerID="cri-o://08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6" gracePeriod=30
Sep 09 20:49:04.455305 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:04.455106    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kube-rbac-proxy" containerID="cri-o://42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace" gracePeriod=30
Sep 09 20:49:04.633510 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:04.633469    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerID="42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace" exitCode=2
Sep 09 20:49:04.633687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:04.633514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerDied","Data":"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"}
Sep 09 20:49:06.175649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.175611    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:49:06.178805 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.178780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.181425 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.181399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 20:49:06.181425 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.181423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:49:06.188245 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.188221    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:49:06.191957 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.191937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:49:06.192067 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.192044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.194578 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.194555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:49:06.205649 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.205623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:49:06.273149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskrc\" (UniqueName: \"kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w45f9\" (UniqueName: \"kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.273327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.273590 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.273360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.373774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.373774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskrc\" (UniqueName: \"kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:06.373894    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:06.373975    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls podName:3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:06.873953611 +0000 UTC m=+2279.710268066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls") pod "isvc-canary-create-predictor-5f7b4999bf-c7ckn" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:06.373975    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.373993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w45f9\" (UniqueName: \"kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374017 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:06.374014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls podName:140cba25-4baf-49d9-b823-e3e5e1aab310 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:06.874003397 +0000 UTC m=+2279.710317839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls") pod "isvc-canary-create-v2-predictor-559cdfb684-5nq9x" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.374556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.374949 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.375009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.374928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.375065 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.375007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.382312 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.382284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w45f9\" (UniqueName: \"kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.383208 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.383186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskrc\" (UniqueName: \"kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.607164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.607056    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:06.607573 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.607538    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:06.879752 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.879656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.879752 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.879712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:06.882231 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.882203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-c7ckn\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:06.882385 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:06.882343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-5nq9x\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:07.090022 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.089983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:07.102477 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.102448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:07.364006 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.363979    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:49:07.365866 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:49:07.365831    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140cba25_4baf_49d9_b823_e3e5e1aab310.slice/crio-2c0f43aef3e9693b2586457cdb832181c9b2c1286b3dcf87ab8d5a6c863462a7 WatchSource:0}: Error finding container 2c0f43aef3e9693b2586457cdb832181c9b2c1286b3dcf87ab8d5a6c863462a7: Status 404 returned error can't find the container with id 2c0f43aef3e9693b2586457cdb832181c9b2c1286b3dcf87ab8d5a6c863462a7
Sep 09 20:49:07.406657 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.406567    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:49:07.409038 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:49:07.409011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0351b9_c88b_45c5_a8b0_ffd2f6f9e6d0.slice/crio-88b8b3a173751d908775fad1b12aea8fab422ee5244d53cfb3207b0587ba5508 WatchSource:0}: Error finding container 88b8b3a173751d908775fad1b12aea8fab422ee5244d53cfb3207b0587ba5508: Status 404 returned error can't find the container with id 88b8b3a173751d908775fad1b12aea8fab422ee5244d53cfb3207b0587ba5508
Sep 09 20:49:07.650861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.650815    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerStarted","Data":"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"}
Sep 09 20:49:07.651070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.650893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerStarted","Data":"88b8b3a173751d908775fad1b12aea8fab422ee5244d53cfb3207b0587ba5508"}
Sep 09 20:49:07.653123 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.653030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerStarted","Data":"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"}
Sep 09 20:49:07.653123 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:07.653083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerStarted","Data":"2c0f43aef3e9693b2586457cdb832181c9b2c1286b3dcf87ab8d5a6c863462a7"}
Sep 09 20:49:08.466394 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:08.466351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 09 20:49:08.471889 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:08.471845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 09 20:49:09.633292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.633256    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:49:09.637096 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.637066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.639501 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.639476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 20:49:09.639625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.639499    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 20:49:09.649300 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.649271    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:49:09.708993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.708950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.709209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.708996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.709209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.709061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4vt\" (UniqueName: \"kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.709209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.709123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.709209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.709145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.810502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.810450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.810502 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.810509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.810761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.810558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4vt\" (UniqueName: \"kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.810761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.810586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.810761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.810710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.811056 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.811035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.811290 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.811272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.811472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.811453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.813246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.813225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.818876 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.818850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4vt\" (UniqueName: \"kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:09.947009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:09.946966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:10.222984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:10.222911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:49:10.225435 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:49:10.225404    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5effc5c3_8e7d_4dba_b767_03f56962b0d8.slice/crio-a34ed8b09fd6745c04cd4a92f1eb98546ccf8ad6ade9347f17000f6755f4ce4b WatchSource:0}: Error finding container a34ed8b09fd6745c04cd4a92f1eb98546ccf8ad6ade9347f17000f6755f4ce4b: Status 404 returned error can't find the container with id a34ed8b09fd6745c04cd4a92f1eb98546ccf8ad6ade9347f17000f6755f4ce4b
Sep 09 20:49:10.667505 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:10.667469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerStarted","Data":"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"}
Sep 09 20:49:10.668010 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:10.667536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerStarted","Data":"a34ed8b09fd6745c04cd4a92f1eb98546ccf8ad6ade9347f17000f6755f4ce4b"}
Sep 09 20:49:11.309643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.309607    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:49:11.426498 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h95k\" (UniqueName: \"kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k\") pod \"3ba71c99-0005-4e88-8072-92fc2d1aa996\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") "
Sep 09 20:49:11.426683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location\") pod \"3ba71c99-0005-4e88-8072-92fc2d1aa996\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") "
Sep 09 20:49:11.426683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") pod \"3ba71c99-0005-4e88-8072-92fc2d1aa996\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") "
Sep 09 20:49:11.426683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert\") pod \"3ba71c99-0005-4e88-8072-92fc2d1aa996\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") "
Sep 09 20:49:11.426683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config\") pod \"3ba71c99-0005-4e88-8072-92fc2d1aa996\" (UID: \"3ba71c99-0005-4e88-8072-92fc2d1aa996\") "
Sep 09 20:49:11.426908 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ba71c99-0005-4e88-8072-92fc2d1aa996" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:11.427028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.426999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ba71c99-0005-4e88-8072-92fc2d1aa996" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:11.427172 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.427042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ba71c99-0005-4e88-8072-92fc2d1aa996" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:11.428791 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.428765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k" (OuterVolumeSpecName: "kube-api-access-4h95k") pod "3ba71c99-0005-4e88-8072-92fc2d1aa996" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996"). InnerVolumeSpecName "kube-api-access-4h95k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:11.428888 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.428794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ba71c99-0005-4e88-8072-92fc2d1aa996" (UID: "3ba71c99-0005-4e88-8072-92fc2d1aa996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:11.527961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.527925    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba71c99-0005-4e88-8072-92fc2d1aa996-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:49:11.527961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.527953    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba71c99-0005-4e88-8072-92fc2d1aa996-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:49:11.527961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.527963    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:49:11.527961 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.527971    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:49:11.528237 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.527980    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h95k\" (UniqueName: \"kubernetes.io/projected/3ba71c99-0005-4e88-8072-92fc2d1aa996-kube-api-access-4h95k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:49:11.673867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.673833    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerID="08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6" exitCode=0
Sep 09 20:49:11.674346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.673890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"
Sep 09 20:49:11.674346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.673915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerDied","Data":"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"}
Sep 09 20:49:11.674346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.674000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j" event={"ID":"3ba71c99-0005-4e88-8072-92fc2d1aa996","Type":"ContainerDied","Data":"349a2b142564fd513b1a4dab9e2f1a0e2464cb87330c330f736e942ddc8a28da"}
Sep 09 20:49:11.674346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.674044    2579 scope.go:117] "RemoveContainer" containerID="42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"
Sep 09 20:49:11.675717 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.675695    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerID="c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a" exitCode=0
Sep 09 20:49:11.675829 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.675771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerDied","Data":"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"}
Sep 09 20:49:11.677849 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.677829    2579 generic.go:356] "Generic (PLEG): container finished" podID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerID="e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce" exitCode=0
Sep 09 20:49:11.677947 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.677901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerDied","Data":"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"}
Sep 09 20:49:11.683274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.683255    2579 scope.go:117] "RemoveContainer" containerID="08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"
Sep 09 20:49:11.690955 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.690931    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:49:11.693932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.693911    2579 scope.go:117] "RemoveContainer" containerID="6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"
Sep 09 20:49:11.695117 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.695071    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-kpq2j"]
Sep 09 20:49:11.705232 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.705210    2579 scope.go:117] "RemoveContainer" containerID="42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"
Sep 09 20:49:11.705579 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:11.705559    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace\": container with ID starting with 42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace not found: ID does not exist" containerID="42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"
Sep 09 20:49:11.705645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.705588    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace"} err="failed to get container status \"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace\": rpc error: code = NotFound desc = could not find container \"42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace\": container with ID starting with 42de63413b2f8d42899960f2f3e4b32f49b12e2d02ec97f7ac11b4bd6d948ace not found: ID does not exist"
Sep 09 20:49:11.705645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.705607    2579 scope.go:117] "RemoveContainer" containerID="08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"
Sep 09 20:49:11.705901 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:11.705884    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6\": container with ID starting with 08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6 not found: ID does not exist" containerID="08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"
Sep 09 20:49:11.705959 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.705910    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6"} err="failed to get container status \"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6\": rpc error: code = NotFound desc = could not find container \"08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6\": container with ID starting with 08e679bfb171f310772b1d1321be6ee674093bd131fa53ed957929ce79b89fe6 not found: ID does not exist"
Sep 09 20:49:11.705959 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.705931    2579 scope.go:117] "RemoveContainer" containerID="6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"
Sep 09 20:49:11.706229 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:49:11.706208    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3\": container with ID starting with 6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3 not found: ID does not exist" containerID="6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"
Sep 09 20:49:11.706288 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:11.706236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3"} err="failed to get container status \"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3\": rpc error: code = NotFound desc = could not find container \"6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3\": container with ID starting with 6abd98da34746c33c07b25ff2f8bc2c6f0721b1dd6d26387ed4d2df0e101a4e3 not found: ID does not exist"
Sep 09 20:49:12.685443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.685407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerStarted","Data":"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"}
Sep 09 20:49:12.685929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.685513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerStarted","Data":"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"}
Sep 09 20:49:12.685929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.685832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:12.686050 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.685981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:12.687475 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.687444    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:12.688862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.688841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerStarted","Data":"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"}
Sep 09 20:49:12.688984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.688890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerStarted","Data":"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"}
Sep 09 20:49:12.689174 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.689159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:12.689255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.689199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:12.690170 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.690148    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:12.704929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.704876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podStartSLOduration=2.775380582 podStartE2EDuration="6.704858732s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.92947815s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:12.702983676 +0000 UTC m=+2285.539298142" watchObservedRunningTime="2026-09-09 20:49:12.704858732 +0000 UTC m=+2285.541173194"
Sep 09 20:49:12.720949 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:12.720895    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podStartSLOduration=2.816360043 podStartE2EDuration="6.720877973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90451793s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:12.71985709 +0000 UTC m=+2285.556171556" watchObservedRunningTime="2026-09-09 20:49:12.720877973 +0000 UTC m=+2285.557192435"
Sep 09 20:49:13.612132 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:13.612102    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" path="/var/lib/kubelet/pods/3ba71c99-0005-4e88-8072-92fc2d1aa996/volumes"
Sep 09 20:49:13.696409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:13.696363    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:13.696817 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:13.696476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:14.702546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:14.702512    2579 generic.go:356] "Generic (PLEG): container finished" podID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerID="d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41" exitCode=0
Sep 09 20:49:14.702923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:14.702593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerDied","Data":"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"}
Sep 09 20:49:15.709408 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:15.709372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerStarted","Data":"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"}
Sep 09 20:49:15.709768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:15.709440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerStarted","Data":"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"}
Sep 09 20:49:15.709768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:15.709648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:15.731860 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:15.731805    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" podStartSLOduration=2.8324415910000003 podStartE2EDuration="6.731789119s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899347528s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:15.730528848 +0000 UTC m=+2288.566843333" watchObservedRunningTime="2026-09-09 20:49:15.731789119 +0000 UTC m=+2288.568103581"
Sep 09 20:49:16.607457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:16.607412    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:16.607823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:16.607794    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:16.714038 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:16.713993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:18.700870 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:18.700838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:49:18.701375 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:18.701267    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:49:18.701451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:18.701377    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:18.701718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:18.701694    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:19.611882 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:19.611835    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:19.612308 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:19.612282    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:22.724164 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:22.724133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:28.701585 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:28.701540    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:28.702465 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:28.701725    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:29.612510 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:29.612464    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:29.612984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:29.612962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:38.701788 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:38.701744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:38.702179 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:38.701744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:39.612891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:39.612847    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:39.613449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:39.613418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:48.701564 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:48.701523    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:48.702133 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:48.701697    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:49.612295 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:49.612243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:49.612741 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:49.612721    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:52.776848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:52.776808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:49:58.702022 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:58.701942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:49:58.702488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:58.701935    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:49:59.612507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.612470    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:49:59.612992 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.612962    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:49:59.699431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.699382    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:49:59.699672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.699644    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kserve-container" containerID="cri-o://0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6" gracePeriod=30
Sep 09 20:49:59.699815 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.699684    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kube-rbac-proxy" containerID="cri-o://12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46" gracePeriod=30
Sep 09 20:49:59.913417 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.913386    2579 generic.go:356] "Generic (PLEG): container finished" podID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerID="12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46" exitCode=2
Sep 09 20:49:59.913780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:49:59.913466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerDied","Data":"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"}
Sep 09 20:50:02.719195 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:02.719154    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 09 20:50:02.725656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:02.725605    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.48:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 09 20:50:06.945577 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.945547    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:50:06.947266 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.947242    2579 generic.go:356] "Generic (PLEG): container finished" podID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerID="0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6" exitCode=0
Sep 09 20:50:06.947366 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.947312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerDied","Data":"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"}
Sep 09 20:50:06.947408 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.947380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp" event={"ID":"5effc5c3-8e7d-4dba-b767-03f56962b0d8","Type":"ContainerDied","Data":"a34ed8b09fd6745c04cd4a92f1eb98546ccf8ad6ade9347f17000f6755f4ce4b"}
Sep 09 20:50:06.947443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.947408    2579 scope.go:117] "RemoveContainer" containerID="12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"
Sep 09 20:50:06.955996 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.955974    2579 scope.go:117] "RemoveContainer" containerID="0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"
Sep 09 20:50:06.964774 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.964742    2579 scope.go:117] "RemoveContainer" containerID="d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"
Sep 09 20:50:06.973639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.973618    2579 scope.go:117] "RemoveContainer" containerID="12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"
Sep 09 20:50:06.973964 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:06.973941    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46\": container with ID starting with 12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46 not found: ID does not exist" containerID="12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"
Sep 09 20:50:06.974045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.973974    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46"} err="failed to get container status \"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46\": rpc error: code = NotFound desc = could not find container \"12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46\": container with ID starting with 12f6eb154e518ac4f7ab4992adee0da17c5e48ed96b1ef42738eb1e47ae98e46 not found: ID does not exist"
Sep 09 20:50:06.974045 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.973994    2579 scope.go:117] "RemoveContainer" containerID="0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"
Sep 09 20:50:06.974340 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:06.974322    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6\": container with ID starting with 0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6 not found: ID does not exist" containerID="0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"
Sep 09 20:50:06.974471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.974354    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6"} err="failed to get container status \"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6\": rpc error: code = NotFound desc = could not find container \"0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6\": container with ID starting with 0dbb2565bdb4dbba07386ba2a797c874b65ba3b98d1b40a9261929a3deaf4bd6 not found: ID does not exist"
Sep 09 20:50:06.974471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.974369    2579 scope.go:117] "RemoveContainer" containerID="d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"
Sep 09 20:50:06.974646 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:06.974629    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41\": container with ID starting with d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41 not found: ID does not exist" containerID="d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"
Sep 09 20:50:06.974709 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:06.974649    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41"} err="failed to get container status \"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41\": rpc error: code = NotFound desc = could not find container \"d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41\": container with ID starting with d0eea43b6c6a0d564df547a339e45697a05f4206adf4f6c276ccd63452200d41 not found: ID does not exist"
Sep 09 20:50:07.028551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert\") pod \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") "
Sep 09 20:50:07.028551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b4vt\" (UniqueName: \"kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt\") pod \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") "
Sep 09 20:50:07.028761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location\") pod \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") "
Sep 09 20:50:07.028761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config\") pod \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") "
Sep 09 20:50:07.028761 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls\") pod \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\" (UID: \"5effc5c3-8e7d-4dba-b767-03f56962b0d8\") "
Sep 09 20:50:07.028969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5effc5c3-8e7d-4dba-b767-03f56962b0d8" (UID: "5effc5c3-8e7d-4dba-b767-03f56962b0d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:07.029043 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5effc5c3-8e7d-4dba-b767-03f56962b0d8" (UID: "5effc5c3-8e7d-4dba-b767-03f56962b0d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:07.029043 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.028993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5effc5c3-8e7d-4dba-b767-03f56962b0d8" (UID: "5effc5c3-8e7d-4dba-b767-03f56962b0d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:07.030930 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.030903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5effc5c3-8e7d-4dba-b767-03f56962b0d8" (UID: "5effc5c3-8e7d-4dba-b767-03f56962b0d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:07.031016 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.030967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt" (OuterVolumeSpecName: "kube-api-access-9b4vt") pod "5effc5c3-8e7d-4dba-b767-03f56962b0d8" (UID: "5effc5c3-8e7d-4dba-b767-03f56962b0d8"). InnerVolumeSpecName "kube-api-access-9b4vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:07.129492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.129449    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:07.129492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.129486    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b4vt\" (UniqueName: \"kubernetes.io/projected/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-api-access-9b4vt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:07.129492 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.129497    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:07.129748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.129507    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5effc5c3-8e7d-4dba-b767-03f56962b0d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:07.129748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.129518    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5effc5c3-8e7d-4dba-b767-03f56962b0d8-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:07.953347 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.953319    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"
Sep 09 20:50:07.971071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.971034    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:50:07.975288 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:07.975258    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-vtjqp"]
Sep 09 20:50:08.701655 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:08.701610    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:50:08.701855 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:08.701804    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:50:09.611046 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:09.611008    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" path="/var/lib/kubelet/pods/5effc5c3-8e7d-4dba-b767-03f56962b0d8/volumes"
Sep 09 20:50:09.612341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:09.612316    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:09.612822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:09.612806    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:18.701474 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:18.701437    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:50:18.701833 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:18.701634    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:50:19.612653 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:19.612614    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:19.613127 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:19.613098    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:22.606667 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:22.606612    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 09 20:50:25.606912 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:25.606858    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 09 20:50:29.612377 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:29.612333    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:29.612935 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:29.612912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:30.606595 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:30.606552    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:30.606880 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:30.606858    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:32.607909 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:32.607876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:50:32.716778 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:32.716738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:50:32.717023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:32.716991    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container" containerID="cri-o://c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e" gracePeriod=30
Sep 09 20:50:32.717214 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:32.717120    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kube-rbac-proxy" containerID="cri-o://08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee" gracePeriod=30
Sep 09 20:50:32.718042 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:32.718025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:50:33.067209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:33.067178    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerID="08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee" exitCode=2
Sep 09 20:50:33.067391 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:33.067252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerDied","Data":"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"}
Sep 09 20:50:33.696607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:33.696566    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 09 20:50:37.860248 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.860221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:50:37.891716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.891685    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location\") pod \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") "
Sep 09 20:50:37.891878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.891734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config\") pod \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") "
Sep 09 20:50:37.891878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.891763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") pod \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") "
Sep 09 20:50:37.891878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.891801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert\") pod \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") "
Sep 09 20:50:37.891878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.891832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kskrc\" (UniqueName: \"kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc\") pod \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\" (UID: \"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0\") "
Sep 09 20:50:37.892134 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.892030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:37.892271 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.892237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:37.892271 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.892247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:37.894247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.894215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:37.894360 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.894248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc" (OuterVolumeSpecName: "kube-api-access-kskrc") pod "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" (UID: "3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0"). InnerVolumeSpecName "kube-api-access-kskrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:37.992580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.992492    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:37.992580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.992519    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:37.992580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.992528    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:37.992580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.992538    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:37.992580 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:37.992546    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskrc\" (UniqueName: \"kubernetes.io/projected/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0-kube-api-access-kskrc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:38.093613 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.093576    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerID="c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e" exitCode=0
Sep 09 20:50:38.093814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.093624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerDied","Data":"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"}
Sep 09 20:50:38.093814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.093638    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"
Sep 09 20:50:38.093814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.093674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn" event={"ID":"3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0","Type":"ContainerDied","Data":"88b8b3a173751d908775fad1b12aea8fab422ee5244d53cfb3207b0587ba5508"}
Sep 09 20:50:38.093814 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.093700    2579 scope.go:117] "RemoveContainer" containerID="08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"
Sep 09 20:50:38.103121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.103098    2579 scope.go:117] "RemoveContainer" containerID="c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"
Sep 09 20:50:38.111256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.111234    2579 scope.go:117] "RemoveContainer" containerID="c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"
Sep 09 20:50:38.115245 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.115221    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:50:38.118718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.118691    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-c7ckn"]
Sep 09 20:50:38.122111 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.122071    2579 scope.go:117] "RemoveContainer" containerID="08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"
Sep 09 20:50:38.122630 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:38.122607    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee\": container with ID starting with 08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee not found: ID does not exist" containerID="08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"
Sep 09 20:50:38.122702 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.122645    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee"} err="failed to get container status \"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee\": rpc error: code = NotFound desc = could not find container \"08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee\": container with ID starting with 08e27b3e4cd740f158a287eb75f1b4a6d28782febfc5711a2a61ddffbe8400ee not found: ID does not exist"
Sep 09 20:50:38.122702 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.122672    2579 scope.go:117] "RemoveContainer" containerID="c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"
Sep 09 20:50:38.123041 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:38.123021    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e\": container with ID starting with c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e not found: ID does not exist" containerID="c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"
Sep 09 20:50:38.123103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.123049    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e"} err="failed to get container status \"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e\": rpc error: code = NotFound desc = could not find container \"c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e\": container with ID starting with c84235abe82540c94e721a9a5b650d66ead3b636cbe4e295ae0ea5fab997c36e not found: ID does not exist"
Sep 09 20:50:38.123103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.123072    2579 scope.go:117] "RemoveContainer" containerID="c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"
Sep 09 20:50:38.123382 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:38.123360    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a\": container with ID starting with c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a not found: ID does not exist" containerID="c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"
Sep 09 20:50:38.123448 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:38.123390    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a"} err="failed to get container status \"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a\": rpc error: code = NotFound desc = could not find container \"c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a\": container with ID starting with c0c68a89fc7d9c8e89295d18acdbbda38a4c194d81a6c96f55f8b49a60ee608a not found: ID does not exist"
Sep 09 20:50:39.610548 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:39.610516    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" path="/var/lib/kubelet/pods/3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0/volumes"
Sep 09 20:50:40.606722 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:40.606677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:40.607147 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:40.607126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:46.237075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:46.237032    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:50:46.237904 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:46.237292    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container" containerID="cri-o://2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9" gracePeriod=30
Sep 09 20:50:46.237904 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:46.237311    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kube-rbac-proxy" containerID="cri-o://902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5" gracePeriod=30
Sep 09 20:50:47.134556 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:47.134521    2579 generic.go:356] "Generic (PLEG): container finished" podID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerID="902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5" exitCode=2
Sep 09 20:50:47.134744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:47.134595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerDied","Data":"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"}
Sep 09 20:50:48.697330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:48.697282    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 09 20:50:50.607058 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:50.607010    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:50:50.607564 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:50.607530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:50:51.388464 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.388437    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:50:51.509932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.509836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert\") pod \"140cba25-4baf-49d9-b823-e3e5e1aab310\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") "
Sep 09 20:50:51.509932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.509904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w45f9\" (UniqueName: \"kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9\") pod \"140cba25-4baf-49d9-b823-e3e5e1aab310\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") "
Sep 09 20:50:51.509932 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.509932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") pod \"140cba25-4baf-49d9-b823-e3e5e1aab310\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") "
Sep 09 20:50:51.510264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.509960    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location\") pod \"140cba25-4baf-49d9-b823-e3e5e1aab310\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") "
Sep 09 20:50:51.510264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.509983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config\") pod \"140cba25-4baf-49d9-b823-e3e5e1aab310\" (UID: \"140cba25-4baf-49d9-b823-e3e5e1aab310\") "
Sep 09 20:50:51.510397 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.510367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "140cba25-4baf-49d9-b823-e3e5e1aab310" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:51.510436 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.510383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "140cba25-4baf-49d9-b823-e3e5e1aab310" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:51.510470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.510431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "140cba25-4baf-49d9-b823-e3e5e1aab310" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:51.512346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.512316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "140cba25-4baf-49d9-b823-e3e5e1aab310" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:51.512453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.512414    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9" (OuterVolumeSpecName: "kube-api-access-w45f9") pod "140cba25-4baf-49d9-b823-e3e5e1aab310" (UID: "140cba25-4baf-49d9-b823-e3e5e1aab310"). InnerVolumeSpecName "kube-api-access-w45f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:51.532551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532511    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:50:51.532865 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532852    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="storage-initializer"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532867    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="storage-initializer"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kube-rbac-proxy"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532884    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kube-rbac-proxy"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kserve-container"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532897    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kserve-container"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532902    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kube-rbac-proxy"
Sep 09 20:50:51.532910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532908    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532928    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532934    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532939    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532948    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532953    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532960    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532965    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532972    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532978    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532984    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532990    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.532996    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533001    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533007    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533012    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="storage-initializer"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533059    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533066    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533072    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533077    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533082    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533106    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5effc5c3-8e7d-4dba-b767-03f56962b0d8" containerName="kube-rbac-proxy"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533111    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0351b9-c88b-45c5-a8b0-ffd2f6f9e6d0" containerName="kserve-container"
Sep 09 20:50:51.533136 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.533116    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba71c99-0005-4e88-8072-92fc2d1aa996" containerName="kserve-container"
Sep 09 20:50:51.536346 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.536329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.538847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.538826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:50:51.538847 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.538842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:50:51.544742 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.544715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:50:51.610883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.610829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh5z\" (UniqueName: \"kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.610942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.610971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611152    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w45f9\" (UniqueName: \"kubernetes.io/projected/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-api-access-w45f9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611173    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/140cba25-4baf-49d9-b823-e3e5e1aab310-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611190    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/140cba25-4baf-49d9-b823-e3e5e1aab310-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611204    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.611321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.611221    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/140cba25-4baf-49d9-b823-e3e5e1aab310-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:50:51.712401 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh5z\" (UniqueName: \"kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.712604 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.712604 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.712604 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.712604 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.712838 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:51.712623    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:50:51.712838 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:51.712689    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls podName:2c69ee10-a47e-4d53-b103-da5aac555d98 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:52.21267045 +0000 UTC m=+2385.048984891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls") pod "isvc-canary-promote-predictor-648d7cfc8f-h5db6" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:50:51.712993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.712971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.713193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.713175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.713240 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.713195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:51.721371 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:51.721351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh5z\" (UniqueName: \"kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:52.156347 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.156311    2579 generic.go:356] "Generic (PLEG): container finished" podID="140cba25-4baf-49d9-b823-e3e5e1aab310" containerID="2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9" exitCode=0
Sep 09 20:50:52.156545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.156372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"
Sep 09 20:50:52.156545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.156388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerDied","Data":"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"}
Sep 09 20:50:52.156545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.156448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x" event={"ID":"140cba25-4baf-49d9-b823-e3e5e1aab310","Type":"ContainerDied","Data":"2c0f43aef3e9693b2586457cdb832181c9b2c1286b3dcf87ab8d5a6c863462a7"}
Sep 09 20:50:52.156545 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.156473    2579 scope.go:117] "RemoveContainer" containerID="902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"
Sep 09 20:50:52.165063 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.165044    2579 scope.go:117] "RemoveContainer" containerID="2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"
Sep 09 20:50:52.173460 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.173440    2579 scope.go:117] "RemoveContainer" containerID="e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"
Sep 09 20:50:52.173534 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.173449    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:50:52.177288 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.177261    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-5nq9x"]
Sep 09 20:50:52.181408 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.181390    2579 scope.go:117] "RemoveContainer" containerID="902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"
Sep 09 20:50:52.181691 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:52.181673    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5\": container with ID starting with 902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5 not found: ID does not exist" containerID="902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"
Sep 09 20:50:52.181743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.181700    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5"} err="failed to get container status \"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5\": rpc error: code = NotFound desc = could not find container \"902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5\": container with ID starting with 902cd435bcfd988ca9db50145af52017cd2513783b2df12e67805483881353c5 not found: ID does not exist"
Sep 09 20:50:52.181743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.181719    2579 scope.go:117] "RemoveContainer" containerID="2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"
Sep 09 20:50:52.181994 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:52.181972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9\": container with ID starting with 2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9 not found: ID does not exist" containerID="2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"
Sep 09 20:50:52.182121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.181998    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9"} err="failed to get container status \"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9\": rpc error: code = NotFound desc = could not find container \"2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9\": container with ID starting with 2e6b8574ccd58e2942158500da33b103b2ee64f5c21bebc586b04918c13fccd9 not found: ID does not exist"
Sep 09 20:50:52.182121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.182020    2579 scope.go:117] "RemoveContainer" containerID="e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"
Sep 09 20:50:52.182326 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:50:52.182307    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce\": container with ID starting with e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce not found: ID does not exist" containerID="e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"
Sep 09 20:50:52.182367 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.182332    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce"} err="failed to get container status \"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce\": rpc error: code = NotFound desc = could not find container \"e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce\": container with ID starting with e6b72ea7cea35e73165f84371300c2cb7837b21fcd33dbf1d39050ac4c0d38ce not found: ID does not exist"
Sep 09 20:50:52.216923 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.216884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:52.219595 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.219560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-h5db6\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:52.447787 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.447686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:52.721590 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:52.721507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:50:52.724714 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:50:52.724685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c69ee10_a47e_4d53_b103_da5aac555d98.slice/crio-8915dac2d52b9143fe1984bfe1a031f337f310b0be48bc61ffbc661650237b8f WatchSource:0}: Error finding container 8915dac2d52b9143fe1984bfe1a031f337f310b0be48bc61ffbc661650237b8f: Status 404 returned error can't find the container with id 8915dac2d52b9143fe1984bfe1a031f337f310b0be48bc61ffbc661650237b8f
Sep 09 20:50:53.163275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:53.163235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerStarted","Data":"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"}
Sep 09 20:50:53.163459 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:53.163299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerStarted","Data":"8915dac2d52b9143fe1984bfe1a031f337f310b0be48bc61ffbc661650237b8f"}
Sep 09 20:50:53.611561 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:53.611473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140cba25-4baf-49d9-b823-e3e5e1aab310" path="/var/lib/kubelet/pods/140cba25-4baf-49d9-b823-e3e5e1aab310/volumes"
Sep 09 20:50:57.181687 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:57.181649    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerID="71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e" exitCode=0
Sep 09 20:50:57.182193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:57.181709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerDied","Data":"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"}
Sep 09 20:50:58.188302 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.188270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerStarted","Data":"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"}
Sep 09 20:50:58.188778 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.188340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerStarted","Data":"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"}
Sep 09 20:50:58.188778 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.188647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:58.188934 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.188793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:50:58.190023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.189998    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:50:58.207055 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:58.207000    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podStartSLOduration=3.409173218 podStartE2EDuration="7.206984198s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.79781098s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:58.205589749 +0000 UTC m=+2391.041904223" watchObservedRunningTime="2026-09-09 20:50:58.206984198 +0000 UTC m=+2391.043298661"
Sep 09 20:50:59.192692 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:50:59.192650    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:00.607108 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:00.607049    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:00.607587 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:00.607561    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:04.197258 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:04.197229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:51:04.197755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:04.197726    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:10.607152 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:10.607077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:10.607616 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:10.607592    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:14.198189 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:14.198149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:20.607025 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:20.606982    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:20.607482 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:20.607461    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:24.197738 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:24.197699    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:30.606806 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:30.606715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:30.607332 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:30.607306    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:34.198341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:34.198305    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:40.606716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:40.606672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:40.607252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:40.607150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:44.197722 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:44.197683    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:50.146585 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.146545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:51:50.150106 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.150072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.152714 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.152694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 09 20:51:50.152826 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.152695    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:51:50.159021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.158998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:51:50.184238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.184199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.184409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.184251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjpk\" (UniqueName: \"kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.184409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.184312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.184409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.184360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.184409 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.184389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.285772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.285734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjpk\" (UniqueName: \"kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.285772 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.285773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.285796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.285817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286034 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.285901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.286263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.286605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.286678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.286627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.288521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.288503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.293625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.293605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjpk\" (UniqueName: \"kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.461651 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.461563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:50.607147 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.607077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:50.607602 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.607578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:51:50.736509 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:50.736478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:51:50.738245 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:51:50.738217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb68b00_7ae3_40c8_a024_916b37659713.slice/crio-f13bc0be9af28642846584cb2a33601a026cef9f9851ecf51689a374592e0f6c WatchSource:0}: Error finding container f13bc0be9af28642846584cb2a33601a026cef9f9851ecf51689a374592e0f6c: Status 404 returned error can't find the container with id f13bc0be9af28642846584cb2a33601a026cef9f9851ecf51689a374592e0f6c
Sep 09 20:51:51.418911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:51.418875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerStarted","Data":"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"}
Sep 09 20:51:51.419410 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:51.418955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerStarted","Data":"f13bc0be9af28642846584cb2a33601a026cef9f9851ecf51689a374592e0f6c"}
Sep 09 20:51:54.198202 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:54.198162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:51:55.436178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:55.436144    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb68b00-7ae3-40c8-a024-916b37659713" containerID="5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d" exitCode=0
Sep 09 20:51:55.436575 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:55.436188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerDied","Data":"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"}
Sep 09 20:51:56.442651 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:56.442612    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerStarted","Data":"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"}
Sep 09 20:51:56.443074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:56.442686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerStarted","Data":"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"}
Sep 09 20:51:56.443074 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:56.442902    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:56.462662 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:56.462605    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" podStartSLOduration=2.253922132 podStartE2EDuration="6.462589739s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.208667607s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:56.461410805 +0000 UTC m=+2449.297725268" watchObservedRunningTime="2026-09-09 20:51:56.462589739 +0000 UTC m=+2449.298904202"
Sep 09 20:51:57.447405 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:57.447373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:51:58.606118 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:58.606043    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:51:58.606598 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:51:58.606490    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:03.458255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:03.458221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:52:04.198105 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:04.198054    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 09 20:52:08.606367 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:08.606310    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:08.606779 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:08.606738    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:14.199102 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:14.199055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:52:14.634130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:14.634030    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:52:14.634289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:14.634265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container" containerID="cri-o://d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a" gracePeriod=30
Sep 09 20:52:14.634334 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:14.634310    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kube-rbac-proxy" containerID="cri-o://7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3" gracePeriod=30
Sep 09 20:52:15.527442 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:15.527409    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerID="7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3" exitCode=2
Sep 09 20:52:15.527848 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:15.527483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerDied","Data":"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"}
Sep 09 20:52:18.606216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:18.606160    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:18.606753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:18.606727    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:19.193149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.193070    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 09 20:52:19.785369 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.785342    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:52:19.955880 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.955838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") pod \"2c69ee10-a47e-4d53-b103-da5aac555d98\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") "
Sep 09 20:52:19.955880 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.955884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert\") pod \"2c69ee10-a47e-4d53-b103-da5aac555d98\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") "
Sep 09 20:52:19.956197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.955906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config\") pod \"2c69ee10-a47e-4d53-b103-da5aac555d98\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") "
Sep 09 20:52:19.956197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.955936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lh5z\" (UniqueName: \"kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z\") pod \"2c69ee10-a47e-4d53-b103-da5aac555d98\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") "
Sep 09 20:52:19.956197 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.955980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location\") pod \"2c69ee10-a47e-4d53-b103-da5aac555d98\" (UID: \"2c69ee10-a47e-4d53-b103-da5aac555d98\") "
Sep 09 20:52:19.956412 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.956375    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c69ee10-a47e-4d53-b103-da5aac555d98" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:19.956526 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.956380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c69ee10-a47e-4d53-b103-da5aac555d98" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:19.956526 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.956387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c69ee10-a47e-4d53-b103-da5aac555d98" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:19.958325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.958300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c69ee10-a47e-4d53-b103-da5aac555d98" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:19.958436 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:19.958387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z" (OuterVolumeSpecName: "kube-api-access-4lh5z") pod "2c69ee10-a47e-4d53-b103-da5aac555d98" (UID: "2c69ee10-a47e-4d53-b103-da5aac555d98"). InnerVolumeSpecName "kube-api-access-4lh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:20.056718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.056680    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c69ee10-a47e-4d53-b103-da5aac555d98-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:20.056718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.056714    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:20.056718 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.056724    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:20.056964 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.056733    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh5z\" (UniqueName: \"kubernetes.io/projected/2c69ee10-a47e-4d53-b103-da5aac555d98-kube-api-access-4lh5z\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:20.056964 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.056741    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c69ee10-a47e-4d53-b103-da5aac555d98-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:20.549981 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.549943    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerID="d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a" exitCode=0
Sep 09 20:52:20.550176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.550011    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"
Sep 09 20:52:20.550176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.550026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerDied","Data":"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"}
Sep 09 20:52:20.550176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.550112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6" event={"ID":"2c69ee10-a47e-4d53-b103-da5aac555d98","Type":"ContainerDied","Data":"8915dac2d52b9143fe1984bfe1a031f337f310b0be48bc61ffbc661650237b8f"}
Sep 09 20:52:20.550176 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.550146    2579 scope.go:117] "RemoveContainer" containerID="7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"
Sep 09 20:52:20.565765 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.565730    2579 scope.go:117] "RemoveContainer" containerID="d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"
Sep 09 20:52:20.577757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.577645    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:52:20.577833 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.577802    2579 scope.go:117] "RemoveContainer" containerID="71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"
Sep 09 20:52:20.582512 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.582488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-h5db6"]
Sep 09 20:52:20.586421 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.586404    2579 scope.go:117] "RemoveContainer" containerID="7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"
Sep 09 20:52:20.586677 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:20.586658    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3\": container with ID starting with 7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3 not found: ID does not exist" containerID="7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"
Sep 09 20:52:20.586730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.586685    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3"} err="failed to get container status \"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3\": rpc error: code = NotFound desc = could not find container \"7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3\": container with ID starting with 7fa85111c015df440cdeb9cccdad877a527464151414b67859e6c80af50c23c3 not found: ID does not exist"
Sep 09 20:52:20.586730 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.586702    2579 scope.go:117] "RemoveContainer" containerID="d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"
Sep 09 20:52:20.586966 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:20.586945    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a\": container with ID starting with d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a not found: ID does not exist" containerID="d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"
Sep 09 20:52:20.587041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.586976    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a"} err="failed to get container status \"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a\": rpc error: code = NotFound desc = could not find container \"d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a\": container with ID starting with d9415ed2498c5bd96ed005034f43a7b65888d9faa970f7bf5d8ccb42063c910a not found: ID does not exist"
Sep 09 20:52:20.587041 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.586999    2579 scope.go:117] "RemoveContainer" containerID="71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"
Sep 09 20:52:20.587280 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:20.587262    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e\": container with ID starting with 71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e not found: ID does not exist" containerID="71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"
Sep 09 20:52:20.587321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:20.587285    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e"} err="failed to get container status \"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e\": rpc error: code = NotFound desc = could not find container \"71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e\": container with ID starting with 71610cd7314ba0ea30fabce368ee94a9859dbcf01d91967e66b0587830a9894e not found: ID does not exist"
Sep 09 20:52:21.544759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.544718    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545206    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container"
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545226    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container"
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545263    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="storage-initializer"
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545272    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="storage-initializer"
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545283    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kube-rbac-proxy"
Sep 09 20:52:21.545342 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545293    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kube-rbac-proxy"
Sep 09 20:52:21.545685 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545370    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kserve-container"
Sep 09 20:52:21.545685 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.545384    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" containerName="kube-rbac-proxy"
Sep 09 20:52:21.548633 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.548611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.551152 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.551134    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:52:21.551279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.551174    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:52:21.558428 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.558406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:52:21.611213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.611183    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c69ee10-a47e-4d53-b103-da5aac555d98" path="/var/lib/kubelet/pods/2c69ee10-a47e-4d53-b103-da5aac555d98/volumes"
Sep 09 20:52:21.671901 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.671863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.672130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.671913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ftp\" (UniqueName: \"kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.672130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.671941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.672130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.672031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.672130 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.672080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.772837 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.772794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.772851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ftp\" (UniqueName: \"kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.772873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.772907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773028 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.772930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773308 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.773287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.773610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.773726 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.773642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.776096 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.776059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.781217 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.781188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ftp\" (UniqueName: \"kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:21.859286 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:21.859184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:22.132813 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:22.132786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:52:22.135670 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:52:22.135644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97b9542_822f_47d9_aa4e_46964efc52ed.slice/crio-c80f557b85cd87a072af6e6099d8f5b6c87f0f26080d52d5c3a4b4924d696fd1 WatchSource:0}: Error finding container c80f557b85cd87a072af6e6099d8f5b6c87f0f26080d52d5c3a4b4924d696fd1: Status 404 returned error can't find the container with id c80f557b85cd87a072af6e6099d8f5b6c87f0f26080d52d5c3a4b4924d696fd1
Sep 09 20:52:22.560792 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:22.560758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerStarted","Data":"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"}
Sep 09 20:52:22.561301 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:22.560828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerStarted","Data":"c80f557b85cd87a072af6e6099d8f5b6c87f0f26080d52d5c3a4b4924d696fd1"}
Sep 09 20:52:26.580528 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:26.580485    2579 generic.go:356] "Generic (PLEG): container finished" podID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerID="05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a" exitCode=0
Sep 09 20:52:26.580903 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:26.580540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerDied","Data":"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"}
Sep 09 20:52:27.587591 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:27.587552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerStarted","Data":"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"}
Sep 09 20:52:27.587960 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:27.587621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerStarted","Data":"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"}
Sep 09 20:52:27.587960 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:27.587813    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:27.606371 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:27.606317    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podStartSLOduration=2.834377789 podStartE2EDuration="6.60630189s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771924101s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:27.604463013 +0000 UTC m=+2480.440777477" watchObservedRunningTime="2026-09-09 20:52:27.60630189 +0000 UTC m=+2480.442616353"
Sep 09 20:52:28.592315 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:28.592284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:28.593748 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:28.593718    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:52:28.606894 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:28.606844    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:28.607350 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:28.607328    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:29.596264 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:29.596224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:52:33.476050 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:33.475994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 20:52:34.600500 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:34.600472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:52:34.601003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:34.600972    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:52:38.606557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:38.606511    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:38.606980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:38.606955    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:43.461315 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:43.461284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:52:44.601815 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:44.601763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:52:48.606392 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:48.606347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:48.606886 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:48.606830    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:50.248362 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:50.248326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:52:50.248790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:50.248619    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kserve-container" containerID="cri-o://eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2" gracePeriod=30
Sep 09 20:52:50.248790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:50.248658    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kube-rbac-proxy" containerID="cri-o://ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb" gracePeriod=30
Sep 09 20:52:50.690686 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:50.690654    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb68b00-7ae3-40c8-a024-916b37659713" containerID="ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb" exitCode=2
Sep 09 20:52:50.690850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:50.690728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerDied","Data":"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"}
Sep 09 20:52:53.452878 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:53.452837    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 09 20:52:54.601076 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:54.601036    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:52:58.193430 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.193406    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:52:58.286574 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config\") pod \"adb68b00-7ae3-40c8-a024-916b37659713\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") "
Sep 09 20:52:58.286574 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls\") pod \"adb68b00-7ae3-40c8-a024-916b37659713\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") "
Sep 09 20:52:58.286574 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert\") pod \"adb68b00-7ae3-40c8-a024-916b37659713\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") "
Sep 09 20:52:58.286815 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcjpk\" (UniqueName: \"kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk\") pod \"adb68b00-7ae3-40c8-a024-916b37659713\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") "
Sep 09 20:52:58.286815 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location\") pod \"adb68b00-7ae3-40c8-a024-916b37659713\" (UID: \"adb68b00-7ae3-40c8-a024-916b37659713\") "
Sep 09 20:52:58.286922 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "adb68b00-7ae3-40c8-a024-916b37659713" (UID: "adb68b00-7ae3-40c8-a024-916b37659713"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:58.286977 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.286951    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "adb68b00-7ae3-40c8-a024-916b37659713" (UID: "adb68b00-7ae3-40c8-a024-916b37659713"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:52:58.287069 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.287047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "adb68b00-7ae3-40c8-a024-916b37659713" (UID: "adb68b00-7ae3-40c8-a024-916b37659713"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:58.288892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.288863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adb68b00-7ae3-40c8-a024-916b37659713" (UID: "adb68b00-7ae3-40c8-a024-916b37659713"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:58.288892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.288860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk" (OuterVolumeSpecName: "kube-api-access-kcjpk") pod "adb68b00-7ae3-40c8-a024-916b37659713" (UID: "adb68b00-7ae3-40c8-a024-916b37659713"). InnerVolumeSpecName "kube-api-access-kcjpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:58.387484 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.387439    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb68b00-7ae3-40c8-a024-916b37659713-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:58.387484 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.387476    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:58.387484 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.387491    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb68b00-7ae3-40c8-a024-916b37659713-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:58.387754 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.387505    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb68b00-7ae3-40c8-a024-916b37659713-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:58.387754 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.387519    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjpk\" (UniqueName: \"kubernetes.io/projected/adb68b00-7ae3-40c8-a024-916b37659713-kube-api-access-kcjpk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:52:58.606242 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.606116    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:52:58.606593 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.606568    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:52:58.728898 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.728861    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb68b00-7ae3-40c8-a024-916b37659713" containerID="eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2" exitCode=0
Sep 09 20:52:58.729119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.728936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"
Sep 09 20:52:58.729119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.728947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerDied","Data":"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"}
Sep 09 20:52:58.729119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.729012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz" event={"ID":"adb68b00-7ae3-40c8-a024-916b37659713","Type":"ContainerDied","Data":"f13bc0be9af28642846584cb2a33601a026cef9f9851ecf51689a374592e0f6c"}
Sep 09 20:52:58.729119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.729043    2579 scope.go:117] "RemoveContainer" containerID="ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"
Sep 09 20:52:58.738547 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.738531    2579 scope.go:117] "RemoveContainer" containerID="eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"
Sep 09 20:52:58.746567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.746546    2579 scope.go:117] "RemoveContainer" containerID="5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"
Sep 09 20:52:58.751813 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.751786    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:52:58.755326 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.755301    2579 scope.go:117] "RemoveContainer" containerID="ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"
Sep 09 20:52:58.755664 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:58.755639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb\": container with ID starting with ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb not found: ID does not exist" containerID="ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"
Sep 09 20:52:58.755764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.755676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb"} err="failed to get container status \"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb\": rpc error: code = NotFound desc = could not find container \"ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb\": container with ID starting with ec20d900f0ee4cde66916398b8a4a4a3095c1998d20b1b2afc5b3a8abf5653eb not found: ID does not exist"
Sep 09 20:52:58.755764 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.755700    2579 scope.go:117] "RemoveContainer" containerID="eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"
Sep 09 20:52:58.755997 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:58.755972    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2\": container with ID starting with eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2 not found: ID does not exist" containerID="eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"
Sep 09 20:52:58.756052 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.756009    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2"} err="failed to get container status \"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2\": rpc error: code = NotFound desc = could not find container \"eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2\": container with ID starting with eff80736535e4505232a26d748757892e0a809115bbcfa21be5fcf08d082d3c2 not found: ID does not exist"
Sep 09 20:52:58.756052 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.756031    2579 scope.go:117] "RemoveContainer" containerID="5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"
Sep 09 20:52:58.756404 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:52:58.756385    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d\": container with ID starting with 5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d not found: ID does not exist" containerID="5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"
Sep 09 20:52:58.756449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.756409    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d"} err="failed to get container status \"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d\": rpc error: code = NotFound desc = could not find container \"5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d\": container with ID starting with 5ca6999b12d9f14463dfe34d5f045163eb724e5d1115ceb308e49e1cadba510d not found: ID does not exist"
Sep 09 20:52:58.756625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:58.756608    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ljktz"]
Sep 09 20:52:59.610910 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:52:59.610875    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb68b00-7ae3-40c8-a024-916b37659713" path="/var/lib/kubelet/pods/adb68b00-7ae3-40c8-a024-916b37659713/volumes"
Sep 09 20:53:04.600972 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:04.600929    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:53:08.606417 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:08.606360    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:08.606874 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:08.606578    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:08.606874 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:08.606834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:08.607434 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:08.607414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:14.601221 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:14.601182    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:53:18.607587 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:18.607542    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:18.608054 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:18.608026    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:24.601186 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:24.601144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:53:28.607423 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:28.607377    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:28.607918 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:28.607894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:34.600924 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:34.600883    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:53:38.607383 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:38.607314    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:38.607790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:38.607765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:44.601857 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:44.601818    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:53:48.607352 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:48.607302    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:48.607790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:48.607763    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:51.649203 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:51.649168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:53:51.649652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:51.649449    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" containerID="cri-o://03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375" gracePeriod=30
Sep 09 20:53:51.649652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:51.649523    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kube-rbac-proxy" containerID="cri-o://f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0" gracePeriod=30
Sep 09 20:53:51.952683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:51.952597    2579 generic.go:356] "Generic (PLEG): container finished" podID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerID="f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0" exitCode=2
Sep 09 20:53:51.952683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:51.952637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerDied","Data":"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"}
Sep 09 20:53:54.597207 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:54.597161    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 09 20:53:54.601481 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:54.601447    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 09 20:53:56.691443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.691416    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:53:56.786282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location\") pod \"d97b9542-822f-47d9-aa4e-46964efc52ed\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") "
Sep 09 20:53:56.786282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config\") pod \"d97b9542-822f-47d9-aa4e-46964efc52ed\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") "
Sep 09 20:53:56.786520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert\") pod \"d97b9542-822f-47d9-aa4e-46964efc52ed\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") "
Sep 09 20:53:56.786520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls\") pod \"d97b9542-822f-47d9-aa4e-46964efc52ed\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") "
Sep 09 20:53:56.786520 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2ftp\" (UniqueName: \"kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp\") pod \"d97b9542-822f-47d9-aa4e-46964efc52ed\" (UID: \"d97b9542-822f-47d9-aa4e-46964efc52ed\") "
Sep 09 20:53:56.786747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d97b9542-822f-47d9-aa4e-46964efc52ed" (UID: "d97b9542-822f-47d9-aa4e-46964efc52ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:56.786747 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d97b9542-822f-47d9-aa4e-46964efc52ed" (UID: "d97b9542-822f-47d9-aa4e-46964efc52ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:56.786883 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.786793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d97b9542-822f-47d9-aa4e-46964efc52ed" (UID: "d97b9542-822f-47d9-aa4e-46964efc52ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:56.788648 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.788622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d97b9542-822f-47d9-aa4e-46964efc52ed" (UID: "d97b9542-822f-47d9-aa4e-46964efc52ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:56.788736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.788651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp" (OuterVolumeSpecName: "kube-api-access-q2ftp") pod "d97b9542-822f-47d9-aa4e-46964efc52ed" (UID: "d97b9542-822f-47d9-aa4e-46964efc52ed"). InnerVolumeSpecName "kube-api-access-q2ftp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:56.869562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869526    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:53:56.869834 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869822    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869837    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869849    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="storage-initializer"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869855    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="storage-initializer"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869866    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="storage-initializer"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869872    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="storage-initializer"
Sep 09 20:53:56.869879 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869884    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869895    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869904    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kserve-container"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869910    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kserve-container"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869956    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kserve-container"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869964    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869973    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kube-rbac-proxy"
Sep 09 20:53:56.870071 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.869979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb68b00-7ae3-40c8-a024-916b37659713" containerName="kserve-container"
Sep 09 20:53:56.873178 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.873162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.875672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.875645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:53:56.875672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.875663    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:53:56.883018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.882994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:53:56.887455 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.887433    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d97b9542-822f-47d9-aa4e-46964efc52ed-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.887571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.887456    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.887571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.887468    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d97b9542-822f-47d9-aa4e-46964efc52ed-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.887571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.887477    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d97b9542-822f-47d9-aa4e-46964efc52ed-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.887571 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.887484    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ftp\" (UniqueName: \"kubernetes.io/projected/d97b9542-822f-47d9-aa4e-46964efc52ed-kube-api-access-q2ftp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:53:56.974573 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.974541    2579 generic.go:356] "Generic (PLEG): container finished" podID="d97b9542-822f-47d9-aa4e-46964efc52ed" containerID="03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375" exitCode=0
Sep 09 20:53:56.974755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.974595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"
Sep 09 20:53:56.974755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.974614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerDied","Data":"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"}
Sep 09 20:53:56.974755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.974673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5" event={"ID":"d97b9542-822f-47d9-aa4e-46964efc52ed","Type":"ContainerDied","Data":"c80f557b85cd87a072af6e6099d8f5b6c87f0f26080d52d5c3a4b4924d696fd1"}
Sep 09 20:53:56.974755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.974701    2579 scope.go:117] "RemoveContainer" containerID="f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"
Sep 09 20:53:56.984149 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.984121    2579 scope.go:117] "RemoveContainer" containerID="03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"
Sep 09 20:53:56.987869 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.987848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qg6\" (UniqueName: \"kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.987980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.987882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.987980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.987907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.987980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.987930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.988166 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.988018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:56.992098 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.992066    2579 scope.go:117] "RemoveContainer" containerID="05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"
Sep 09 20:53:56.996308 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.996285    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:53:56.999983 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:56.999960    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-d4ph5"]
Sep 09 20:53:57.000449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.000436    2579 scope.go:117] "RemoveContainer" containerID="f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"
Sep 09 20:53:57.000747 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:53:57.000713    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0\": container with ID starting with f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0 not found: ID does not exist" containerID="f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"
Sep 09 20:53:57.000812 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.000744    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0"} err="failed to get container status \"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0\": rpc error: code = NotFound desc = could not find container \"f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0\": container with ID starting with f52407c67f06900d917aadfdf8dd9a1264771c41a22d1189ce77197af64b51f0 not found: ID does not exist"
Sep 09 20:53:57.000812 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.000769    2579 scope.go:117] "RemoveContainer" containerID="03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"
Sep 09 20:53:57.001072 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:53:57.001055    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375\": container with ID starting with 03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375 not found: ID does not exist" containerID="03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"
Sep 09 20:53:57.001227 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.001079    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375"} err="failed to get container status \"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375\": rpc error: code = NotFound desc = could not find container \"03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375\": container with ID starting with 03da9f773a2782d26e88df72f245c6e080cdb248866fe6a45ffeca9faa4cf375 not found: ID does not exist"
Sep 09 20:53:57.001227 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.001112    2579 scope.go:117] "RemoveContainer" containerID="05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"
Sep 09 20:53:57.001376 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:53:57.001356    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a\": container with ID starting with 05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a not found: ID does not exist" containerID="05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"
Sep 09 20:53:57.001413 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.001382    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a"} err="failed to get container status \"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a\": rpc error: code = NotFound desc = could not find container \"05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a\": container with ID starting with 05679eaec13d0342bddb1ce085be7c13594eed672074f9f80c4654d42c66c93a not found: ID does not exist"
Sep 09 20:53:57.088861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.088768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.088861 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.088828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.088871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qg6\" (UniqueName: \"kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.088898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.088931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089265 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.089239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.089653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.089937 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.089655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.091607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.091585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.097059 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.097041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qg6\" (UniqueName: \"kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6\") pod \"isvc-canary-rollback-predictor-6477dbf789-fftpd\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.183763 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.183720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:53:57.456783 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.456754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:53:57.458979 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:53:57.458954    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a5a7a4_801d_4a3c_bfb7_a2b0309c5dd0.slice/crio-c16eea5191df9c1c34096ca55afe63c96ff6d48d0231d0202c6a2e9ba2378c62 WatchSource:0}: Error finding container c16eea5191df9c1c34096ca55afe63c96ff6d48d0231d0202c6a2e9ba2378c62: Status 404 returned error can't find the container with id c16eea5191df9c1c34096ca55afe63c96ff6d48d0231d0202c6a2e9ba2378c62
Sep 09 20:53:57.611557 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.611511    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97b9542-822f-47d9-aa4e-46964efc52ed" path="/var/lib/kubelet/pods/d97b9542-822f-47d9-aa4e-46964efc52ed/volumes"
Sep 09 20:53:57.981098 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.981055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerStarted","Data":"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"}
Sep 09 20:53:57.981468 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:57.981154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerStarted","Data":"c16eea5191df9c1c34096ca55afe63c96ff6d48d0231d0202c6a2e9ba2378c62"}
Sep 09 20:53:58.607256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:58.607213    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:53:58.607734 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:53:58.607711    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:01.999150 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:01.999113    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerID="cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81" exitCode=0
Sep 09 20:54:01.999504 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:01.999159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerDied","Data":"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"}
Sep 09 20:54:03.006156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:03.006117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerStarted","Data":"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"}
Sep 09 20:54:03.006510 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:03.006206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerStarted","Data":"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"}
Sep 09 20:54:03.006510 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:03.006388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:54:03.024182 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:03.024134    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podStartSLOduration=3.200765428 podStartE2EDuration="7.024120024s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823354596s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:03.022642173 +0000 UTC m=+2575.858956635" watchObservedRunningTime="2026-09-09 20:54:03.024120024 +0000 UTC m=+2575.860434483"
Sep 09 20:54:04.010249 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:04.010200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:54:04.011330 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:04.011298    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:05.014529 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:05.014494    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:08.607629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:08.607576    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:08.608047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:08.608022    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:10.019782 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:10.019751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:54:10.020389 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:10.020362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:10.606249 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:10.606204    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:10.606719 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:10.606693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:20.021175 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.021134    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:20.606240 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.606193    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:20.606698 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.606671    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:20.667109 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.667056    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:54:20.670682 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.670660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.673150 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.673118    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 09 20:54:20.673150 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.673128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 09 20:54:20.673336 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.673270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:54:20.680036 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.680009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:54:20.694231 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.694204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpbs\" (UniqueName: \"kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.694378 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.694301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.694527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.694505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.694568 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.694543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.795499 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.795463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpbs\" (UniqueName: \"kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.795677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.795542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.795677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.795570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.795677 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.795593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.795835 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:54:20.795696    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 09 20:54:20.795835 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:54:20.795751    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls podName:6cb923a3-86e6-4665-9e42-20adeafabf28 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:21.2957303 +0000 UTC m=+2594.132044743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls") pod "isvc-sklearn-s3-predictor-7487d6f469-dvcqq" (UID: "6cb923a3-86e6-4665-9e42-20adeafabf28") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 09 20:54:20.795984 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.795962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.796276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.796258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:20.804246 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:20.804222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpbs\" (UniqueName: \"kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:21.299659 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:21.299618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:21.302223 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:21.302194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7487d6f469-dvcqq\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:21.581128 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:21.581020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:21.852905 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:21.852873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:54:21.855679 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:54:21.855649    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb923a3_86e6_4665_9e42_20adeafabf28.slice/crio-5abeba3c2d12ff816e04f06df46952fafeb817d80a5ac8eafd8eed3c970dd5ba WatchSource:0}: Error finding container 5abeba3c2d12ff816e04f06df46952fafeb817d80a5ac8eafd8eed3c970dd5ba: Status 404 returned error can't find the container with id 5abeba3c2d12ff816e04f06df46952fafeb817d80a5ac8eafd8eed3c970dd5ba
Sep 09 20:54:22.083292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:22.083253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerStarted","Data":"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"}
Sep 09 20:54:22.083477 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:22.083316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerStarted","Data":"5abeba3c2d12ff816e04f06df46952fafeb817d80a5ac8eafd8eed3c970dd5ba"}
Sep 09 20:54:24.092757 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:24.092719    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerID="09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc" exitCode=0
Sep 09 20:54:24.093167 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:24.092794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerDied","Data":"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"}
Sep 09 20:54:25.099219 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:25.099183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerStarted","Data":"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"}
Sep 09 20:54:25.099696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:25.099247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerStarted","Data":"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"}
Sep 09 20:54:25.099696 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:25.099449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:25.121080 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:25.121024    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podStartSLOduration=3.889971383 podStartE2EDuration="5.121005942s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.231034559s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:25.118639256 +0000 UTC m=+2597.954953718" watchObservedRunningTime="2026-09-09 20:54:25.121005942 +0000 UTC m=+2597.957320404"
Sep 09 20:54:26.104576 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:26.104540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:26.105899 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:26.105864    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:54:27.108527 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:27.108485    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:54:30.021079 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:30.020995    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:30.606570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:30.606522    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:30.607020 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:30.606990    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:32.113050 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:32.113019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:54:32.113664 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:32.113637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:54:40.020797 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:40.020744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:40.607177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:40.607117    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:40.607590 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:40.607564    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:42.114283 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:42.114245    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:54:50.020990 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:50.020950    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:54:50.606944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:50.606894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:54:50.607440 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:50.607410    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:54:52.114607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:54:52.114570    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:00.020327 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:00.020276    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:55:00.606494 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:00.606442    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:00.606956 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:00.606929    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:02.114592 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:02.114553    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:10.020575 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:10.020533    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:55:10.606822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:10.606768    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:10.607313 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:10.607288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:11.606052 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:11.606004    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 09 20:55:12.113655 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:12.113616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:18.580098 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:18.580037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:55:18.580550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:18.580294    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container" containerID="cri-o://777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df" gracePeriod=30
Sep 09 20:55:18.580550 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:18.580378    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kube-rbac-proxy" containerID="cri-o://a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493" gracePeriod=30
Sep 09 20:55:18.581289 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:18.581264    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:55:19.335293 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:19.335255    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerID="a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493" exitCode=2
Sep 09 20:55:19.335441 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:19.335329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerDied","Data":"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"}
Sep 09 20:55:20.015391 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:20.015345    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 09 20:55:20.606559 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:20.606513    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:20.606978 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:20.606952    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:22.114066 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:22.114027    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:23.747243 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.747215    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:55:23.841801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.841716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location\") pod \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") "
Sep 09 20:55:23.841801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.841751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config\") pod \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") "
Sep 09 20:55:23.841801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.841784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert\") pod \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") "
Sep 09 20:55:23.842114 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.841828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls\") pod \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") "
Sep 09 20:55:23.842114 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.841874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87qg6\" (UniqueName: \"kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6\") pod \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\" (UID: \"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0\") "
Sep 09 20:55:23.842238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.842143    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" (UID: "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:23.842238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.842180    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" (UID: "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:23.842238 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.842203    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" (UID: "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:23.844213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.844184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" (UID: "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:23.844381 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.844270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6" (OuterVolumeSpecName: "kube-api-access-87qg6") pod "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" (UID: "d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0"). InnerVolumeSpecName "kube-api-access-87qg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:23.942546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.942505    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:23.942546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.942540    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:23.942546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.942548    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qg6\" (UniqueName: \"kubernetes.io/projected/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-api-access-87qg6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:23.942546 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.942557    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:23.942806 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:23.942568    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:24.360118 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.360065    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerID="777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df" exitCode=0
Sep 09 20:55:24.360310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.360150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"
Sep 09 20:55:24.360310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.360148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerDied","Data":"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"}
Sep 09 20:55:24.360310 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.360293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd" event={"ID":"d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0","Type":"ContainerDied","Data":"c16eea5191df9c1c34096ca55afe63c96ff6d48d0231d0202c6a2e9ba2378c62"}
Sep 09 20:55:24.360489 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.360324    2579 scope.go:117] "RemoveContainer" containerID="a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"
Sep 09 20:55:24.369564 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.369544    2579 scope.go:117] "RemoveContainer" containerID="777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"
Sep 09 20:55:24.378038 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.378014    2579 scope.go:117] "RemoveContainer" containerID="cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"
Sep 09 20:55:24.381924 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.381893    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:55:24.385675 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.385653    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-fftpd"]
Sep 09 20:55:24.386652 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.386635    2579 scope.go:117] "RemoveContainer" containerID="a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"
Sep 09 20:55:24.386945 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:24.386920    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493\": container with ID starting with a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493 not found: ID does not exist" containerID="a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"
Sep 09 20:55:24.387005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.386951    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493"} err="failed to get container status \"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493\": rpc error: code = NotFound desc = could not find container \"a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493\": container with ID starting with a8c805610a4773555cc0d98391f947d79528ec09b207f77460e18ae43cc8e493 not found: ID does not exist"
Sep 09 20:55:24.387005 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.386972    2579 scope.go:117] "RemoveContainer" containerID="777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"
Sep 09 20:55:24.387257 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:24.387237    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df\": container with ID starting with 777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df not found: ID does not exist" containerID="777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"
Sep 09 20:55:24.387337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.387269    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df"} err="failed to get container status \"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df\": rpc error: code = NotFound desc = could not find container \"777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df\": container with ID starting with 777667b03c29f1905c95da686516355992692b436d6315c2f8287e390d4866df not found: ID does not exist"
Sep 09 20:55:24.387337 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.387292    2579 scope.go:117] "RemoveContainer" containerID="cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"
Sep 09 20:55:24.387535 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:24.387510    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81\": container with ID starting with cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81 not found: ID does not exist" containerID="cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"
Sep 09 20:55:24.387586 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.387539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81"} err="failed to get container status \"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81\": rpc error: code = NotFound desc = could not find container \"cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81\": container with ID starting with cb10de09881cd9bf6536c83cd21545ec90db13fe73818943a168a32a8ecdda81 not found: ID does not exist"
Sep 09 20:55:24.606111 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.606051    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:24.606488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:24.606468    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:25.610921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:25.610886    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" path="/var/lib/kubelet/pods/d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0/volumes"
Sep 09 20:55:26.896343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896300    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container"
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896695    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container"
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896710    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="storage-initializer"
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896716    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="storage-initializer"
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896726    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kube-rbac-proxy"
Sep 09 20:55:26.896736 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896731    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kube-rbac-proxy"
Sep 09 20:55:26.896937 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896785    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kserve-container"
Sep 09 20:55:26.896937 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.896799    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a5a7a4-801d-4a3c-bfb7-a2b0309c5dd0" containerName="kube-rbac-proxy"
Sep 09 20:55:26.901666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.901645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:26.904060 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.904027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:55:26.904209 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.904120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:55:26.909714 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.909692    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:55:26.966427 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.966394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:26.966640 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.966436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:26.966640 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.966461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4nd\" (UniqueName: \"kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:26.966640 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.966579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:26.966640 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:26.966614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.067746 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.067706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.067746 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.067747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068007 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.067788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068007 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.067812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068007 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.067835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4nd\" (UniqueName: \"kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.068225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068521 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.068498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.068618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.068580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.070435 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.070416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.076615 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.076584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4nd\" (UniqueName: \"kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd\") pod \"isvc-canary-rollback-predictor-6477dbf789-nc2qn\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.212668 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.212581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:27.485683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:27.485654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:55:27.488269 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:55:27.488231    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaa0870_05a1_439c_b68f_42e5eaf1c500.slice/crio-b0fe3b67b7ac840726cd77d55ba5554fe9138cfd4d4d9880394dceb252e7e120 WatchSource:0}: Error finding container b0fe3b67b7ac840726cd77d55ba5554fe9138cfd4d4d9880394dceb252e7e120: Status 404 returned error can't find the container with id b0fe3b67b7ac840726cd77d55ba5554fe9138cfd4d4d9880394dceb252e7e120
Sep 09 20:55:28.381567 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:28.381527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerStarted","Data":"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"}
Sep 09 20:55:28.381953 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:28.381591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerStarted","Data":"b0fe3b67b7ac840726cd77d55ba5554fe9138cfd4d4d9880394dceb252e7e120"}
Sep 09 20:55:32.114629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:32.114588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:32.403160 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:32.403129    2579 generic.go:356] "Generic (PLEG): container finished" podID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerID="813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c" exitCode=0
Sep 09 20:55:32.403347 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:32.403205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerDied","Data":"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"}
Sep 09 20:55:33.410105 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.410049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerStarted","Data":"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"}
Sep 09 20:55:33.410507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.410142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerStarted","Data":"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"}
Sep 09 20:55:33.410507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.410438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:33.410507 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.410495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:33.411896 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.411867    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:55:33.429700 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:33.429652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podStartSLOduration=3.572475837 podStartE2EDuration="7.429636229s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857160392s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:33.428238907 +0000 UTC m=+2666.264553370" watchObservedRunningTime="2026-09-09 20:55:33.429636229 +0000 UTC m=+2666.265950689"
Sep 09 20:55:34.415193 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:34.415147    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:55:34.606912 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:34.606873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:34.607363 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:34.607335    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:39.420262 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:39.420229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:55:39.420755 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:39.420728    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:55:42.114159 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:42.114123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:55:44.606343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:44.606303    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:44.606840 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:44.606748    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:49.421226 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:49.421184    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:55:50.756055 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:50.756022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:55:50.756469 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:50.756298    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" containerID="cri-o://412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225" gracePeriod=30
Sep 09 20:55:50.756469 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:50.756384    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kube-rbac-proxy" containerID="cri-o://a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba" gracePeriod=30
Sep 09 20:55:51.493101 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:51.493047    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerID="a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba" exitCode=2
Sep 09 20:55:51.493284 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:51.493115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerDied","Data":"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"}
Sep 09 20:55:52.109463 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:52.109422    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 09 20:55:52.113822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:52.113795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 09 20:55:54.606061 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:54.606017    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:55:54.606578 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:54.606555    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:55:55.899982 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:55.899955    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:55:56.019639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.019532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwpbs\" (UniqueName: \"kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs\") pod \"6cb923a3-86e6-4665-9e42-20adeafabf28\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") "
Sep 09 20:55:56.019639 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.019588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location\") pod \"6cb923a3-86e6-4665-9e42-20adeafabf28\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") "
Sep 09 20:55:56.019854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.019705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config\") pod \"6cb923a3-86e6-4665-9e42-20adeafabf28\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") "
Sep 09 20:55:56.019854 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.019763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") pod \"6cb923a3-86e6-4665-9e42-20adeafabf28\" (UID: \"6cb923a3-86e6-4665-9e42-20adeafabf28\") "
Sep 09 20:55:56.019938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.019916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cb923a3-86e6-4665-9e42-20adeafabf28" (UID: "6cb923a3-86e6-4665-9e42-20adeafabf28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:55:56.020047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.020026    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb923a3-86e6-4665-9e42-20adeafabf28-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:56.020047 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.020030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6cb923a3-86e6-4665-9e42-20adeafabf28" (UID: "6cb923a3-86e6-4665-9e42-20adeafabf28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:55:56.021946 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.021922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs" (OuterVolumeSpecName: "kube-api-access-pwpbs") pod "6cb923a3-86e6-4665-9e42-20adeafabf28" (UID: "6cb923a3-86e6-4665-9e42-20adeafabf28"). InnerVolumeSpecName "kube-api-access-pwpbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:55:56.022030 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.021977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6cb923a3-86e6-4665-9e42-20adeafabf28" (UID: "6cb923a3-86e6-4665-9e42-20adeafabf28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:55:56.120913 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.120874    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:56.120913 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.120907    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb923a3-86e6-4665-9e42-20adeafabf28-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:56.120913 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.120920    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpbs\" (UniqueName: \"kubernetes.io/projected/6cb923a3-86e6-4665-9e42-20adeafabf28-kube-api-access-pwpbs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:55:56.514672 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.514640    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerID="412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225" exitCode=0
Sep 09 20:55:56.514892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.514708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"
Sep 09 20:55:56.514892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.514718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerDied","Data":"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"}
Sep 09 20:55:56.514892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.514802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq" event={"ID":"6cb923a3-86e6-4665-9e42-20adeafabf28","Type":"ContainerDied","Data":"5abeba3c2d12ff816e04f06df46952fafeb817d80a5ac8eafd8eed3c970dd5ba"}
Sep 09 20:55:56.514892 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.514841    2579 scope.go:117] "RemoveContainer" containerID="a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"
Sep 09 20:55:56.524646 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.524621    2579 scope.go:117] "RemoveContainer" containerID="412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"
Sep 09 20:55:56.532477 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.532457    2579 scope.go:117] "RemoveContainer" containerID="09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"
Sep 09 20:55:56.536551 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.536524    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:55:56.541802 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.541781    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7487d6f469-dvcqq"]
Sep 09 20:55:56.541893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.541818    2579 scope.go:117] "RemoveContainer" containerID="a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"
Sep 09 20:55:56.542196 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:56.542174    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba\": container with ID starting with a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba not found: ID does not exist" containerID="a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"
Sep 09 20:55:56.542287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.542202    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba"} err="failed to get container status \"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba\": rpc error: code = NotFound desc = could not find container \"a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba\": container with ID starting with a00cdc7072d111c4d54c1f1a2c3575594c97fd18a252c6e28b43f8199b4096ba not found: ID does not exist"
Sep 09 20:55:56.542287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.542221    2579 scope.go:117] "RemoveContainer" containerID="412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"
Sep 09 20:55:56.542486 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:56.542469    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225\": container with ID starting with 412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225 not found: ID does not exist" containerID="412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"
Sep 09 20:55:56.542524 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.542493    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225"} err="failed to get container status \"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225\": rpc error: code = NotFound desc = could not find container \"412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225\": container with ID starting with 412752ae38c8188d4a67fce18048883803cd56c15f68c77f93d7c17cbd2dd225 not found: ID does not exist"
Sep 09 20:55:56.542524 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.542508    2579 scope.go:117] "RemoveContainer" containerID="09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"
Sep 09 20:55:56.542784 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:55:56.542767    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc\": container with ID starting with 09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc not found: ID does not exist" containerID="09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"
Sep 09 20:55:56.542822 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:56.542791    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc"} err="failed to get container status \"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc\": rpc error: code = NotFound desc = could not find container \"09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc\": container with ID starting with 09bfc3e87772ae8dfc3309f3d6c98aad62acdf88fd0654859229a7df20bc4dfc not found: ID does not exist"
Sep 09 20:55:57.616251 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:57.616208    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" path="/var/lib/kubelet/pods/6cb923a3-86e6-4665-9e42-20adeafabf28/volumes"
Sep 09 20:55:59.420915 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:55:59.420834    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:56:04.606816 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:04.606754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:04.607274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:04.607248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:09.421433 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:09.421390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:56:14.606850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:14.606795    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:14.607319 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:14.607262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:19.421629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:19.421589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:56:24.606408 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:24.606350    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:24.606865 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:24.606841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:26.606699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:26.606648    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:26.607159 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:26.606974    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:29.420726 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:29.420684    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:56:36.607156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:36.607115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:36.607655 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:36.607625    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:39.421057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:39.421018    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:56:46.606674 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:46.606628    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:46.607207 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:46.607180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:49.422260 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:49.422224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:56:56.606678 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:56.606623    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:56:56.607165 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:56.607013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:56:57.020216 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:57.020182    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:56:57.020469 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:57.020438    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" containerID="cri-o://99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066" gracePeriod=30
Sep 09 20:56:57.020601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:57.020487    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kube-rbac-proxy" containerID="cri-o://6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8" gracePeriod=30
Sep 09 20:56:57.773392 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:57.773356    2579 generic.go:356] "Generic (PLEG): container finished" podID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerID="6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8" exitCode=2
Sep 09 20:56:57.773786 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:57.773434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerDied","Data":"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"}
Sep 09 20:56:59.416004 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:59.415958    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 09 20:56:59.421335 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:56:59.421306    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 09 20:57:01.964716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:01.964688    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:57:02.076309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config\") pod \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") "
Sep 09 20:57:02.076309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert\") pod \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") "
Sep 09 20:57:02.076309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp4nd\" (UniqueName: \"kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd\") pod \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") "
Sep 09 20:57:02.076562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location\") pod \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") "
Sep 09 20:57:02.076562 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls\") pod \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\" (UID: \"dcaa0870-05a1-439c-b68f-42e5eaf1c500\") "
Sep 09 20:57:02.076744 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dcaa0870-05a1-439c-b68f-42e5eaf1c500" (UID: "dcaa0870-05a1-439c-b68f-42e5eaf1c500"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:02.076798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcaa0870-05a1-439c-b68f-42e5eaf1c500" (UID: "dcaa0870-05a1-439c-b68f-42e5eaf1c500"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:02.076798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.076730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcaa0870-05a1-439c-b68f-42e5eaf1c500" (UID: "dcaa0870-05a1-439c-b68f-42e5eaf1c500"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:02.078714 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.078694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcaa0870-05a1-439c-b68f-42e5eaf1c500" (UID: "dcaa0870-05a1-439c-b68f-42e5eaf1c500"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:02.078798 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.078711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd" (OuterVolumeSpecName: "kube-api-access-rp4nd") pod "dcaa0870-05a1-439c-b68f-42e5eaf1c500" (UID: "dcaa0870-05a1-439c-b68f-42e5eaf1c500"). InnerVolumeSpecName "kube-api-access-rp4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:02.177255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.177222    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.177255 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.177253    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcaa0870-05a1-439c-b68f-42e5eaf1c500-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.177453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.177264    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.177453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.177273    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcaa0870-05a1-439c-b68f-42e5eaf1c500-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.177453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.177283    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4nd\" (UniqueName: \"kubernetes.io/projected/dcaa0870-05a1-439c-b68f-42e5eaf1c500-kube-api-access-rp4nd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:02.282419 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:57:02.282694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282680    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="storage-initializer"
Sep 09 20:57:02.282694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282695    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="storage-initializer"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282706    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282711    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282721    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kube-rbac-proxy"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282727    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kube-rbac-proxy"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282739    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="storage-initializer"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282760    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="storage-initializer"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282766    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kube-rbac-proxy"
Sep 09 20:57:02.282790 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282771    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kube-rbac-proxy"
Sep 09 20:57:02.283103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kserve-container"
Sep 09 20:57:02.283103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282824    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kserve-container"
Sep 09 20:57:02.283103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282831    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb923a3-86e6-4665-9e42-20adeafabf28" containerName="kube-rbac-proxy"
Sep 09 20:57:02.283103 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.282837    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerName="kube-rbac-proxy"
Sep 09 20:57:02.286097 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.286069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.289009 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.288984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:57:02.289768 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.289696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:57:02.301702 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.301675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:57:02.379929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.379844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.379929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.379893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.380156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.379976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckzfc\" (UniqueName: \"kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.380156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.380024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.380156 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.380041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.480745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.480705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.480745 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.480752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.480786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckzfc\" (UniqueName: \"kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.480809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481003 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.480826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481252 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.481230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481523 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.481500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.481588 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.481540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.483517 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.483491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.490453 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.490424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckzfc\" (UniqueName: \"kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-hwv6z\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.597220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.597161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:02.796449 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.796406    2579 generic.go:356] "Generic (PLEG): container finished" podID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" containerID="99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066" exitCode=0
Sep 09 20:57:02.796614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.796472    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"
Sep 09 20:57:02.796614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.796485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerDied","Data":"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"}
Sep 09 20:57:02.796614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.796552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn" event={"ID":"dcaa0870-05a1-439c-b68f-42e5eaf1c500","Type":"ContainerDied","Data":"b0fe3b67b7ac840726cd77d55ba5554fe9138cfd4d4d9880394dceb252e7e120"}
Sep 09 20:57:02.796614 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.796587    2579 scope.go:117] "RemoveContainer" containerID="6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"
Sep 09 20:57:02.805498 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.805478    2579 scope.go:117] "RemoveContainer" containerID="99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"
Sep 09 20:57:02.813275 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.813259    2579 scope.go:117] "RemoveContainer" containerID="813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"
Sep 09 20:57:02.821204 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.821187    2579 scope.go:117] "RemoveContainer" containerID="6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"
Sep 09 20:57:02.821668 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:02.821642    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8\": container with ID starting with 6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8 not found: ID does not exist" containerID="6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"
Sep 09 20:57:02.821781 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.821683    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8"} err="failed to get container status \"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8\": rpc error: code = NotFound desc = could not find container \"6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8\": container with ID starting with 6e13b9a49df9ce8ee04212f2a8df87de86e599ca3517f7c213a623ec0fd77be8 not found: ID does not exist"
Sep 09 20:57:02.821781 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.821708    2579 scope.go:117] "RemoveContainer" containerID="99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"
Sep 09 20:57:02.822446 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:02.822421    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066\": container with ID starting with 99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066 not found: ID does not exist" containerID="99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"
Sep 09 20:57:02.822515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.822453    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066"} err="failed to get container status \"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066\": rpc error: code = NotFound desc = could not find container \"99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066\": container with ID starting with 99ceeb1cd49762a27a5e16835dc008198422504838a91187dea2211bb3df5066 not found: ID does not exist"
Sep 09 20:57:02.822515 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.822469    2579 scope.go:117] "RemoveContainer" containerID="813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"
Sep 09 20:57:02.822794 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:02.822770    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c\": container with ID starting with 813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c not found: ID does not exist" containerID="813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"
Sep 09 20:57:02.822882 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.822804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c"} err="failed to get container status \"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c\": rpc error: code = NotFound desc = could not find container \"813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c\": container with ID starting with 813de35f84ac41c5c0ffb0458ca515aaa0d89ea4545e497e33385c6961ad663c not found: ID does not exist"
Sep 09 20:57:02.824292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.824272    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:57:02.834634 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.834611    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-nc2qn"]
Sep 09 20:57:02.870423 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:02.870390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:57:02.872799 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:57:02.872768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ee9206_a125_4b12_a049_a4d2c93be418.slice/crio-78f89b67a9cc25b3a9e5b746ebd83c15151d679efd42adf6ccba20808f4a5b85 WatchSource:0}: Error finding container 78f89b67a9cc25b3a9e5b746ebd83c15151d679efd42adf6ccba20808f4a5b85: Status 404 returned error can't find the container with id 78f89b67a9cc25b3a9e5b746ebd83c15151d679efd42adf6ccba20808f4a5b85
Sep 09 20:57:03.611693 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:03.611660    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaa0870-05a1-439c-b68f-42e5eaf1c500" path="/var/lib/kubelet/pods/dcaa0870-05a1-439c-b68f-42e5eaf1c500/volumes"
Sep 09 20:57:03.802770 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:03.802731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerStarted","Data":"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"}
Sep 09 20:57:03.802995 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:03.802806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerStarted","Data":"78f89b67a9cc25b3a9e5b746ebd83c15151d679efd42adf6ccba20808f4a5b85"}
Sep 09 20:57:06.606694 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:06.606650    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:06.607180 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:06.607143    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:07.821119 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:07.820996    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerID="5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e" exitCode=0
Sep 09 20:57:07.822607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:07.821656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerDied","Data":"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"}
Sep 09 20:57:08.827490 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:08.827451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerStarted","Data":"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"}
Sep 09 20:57:08.827957 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:08.827529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerStarted","Data":"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"}
Sep 09 20:57:08.827957 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:08.827756    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:08.847622 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:08.847561    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podStartSLOduration=2.710404532 podStartE2EDuration="6.847543284s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.137138752s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:08.845947911 +0000 UTC m=+2761.682262405" watchObservedRunningTime="2026-09-09 20:57:08.847543284 +0000 UTC m=+2761.683857747"
Sep 09 20:57:09.831400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:09.831363    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:09.832829 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:09.832801    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:10.834938 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:10.834894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:15.840482 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:15.840455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:57:15.840936 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:15.840907    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:16.606704 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:16.606660    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:16.607185 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:16.607146    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:22.121338 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.121303    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:22.124372 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.124349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.127287 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.127263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 09 20:57:22.127431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.127263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:57:22.127716 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.127698    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 09 20:57:22.133194 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.133168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.133309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.133201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.133309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.133224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdsc\" (UniqueName: \"kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.133309 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.133299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.140222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.140199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:22.234022 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.233980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.234274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.234062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.234274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.234110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.234274 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.234131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdsc\" (UniqueName: \"kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.234274 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:22.234159    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 09 20:57:22.234274 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:22.234230    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls podName:6c3e06a6-be77-43a9-8152-3192b2f584d9 nodeName:}" failed. No retries permitted until 2026-09-09 20:57:22.734211974 +0000 UTC m=+2775.570526416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" (UID: "6c3e06a6-be77-43a9-8152-3192b2f584d9") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 09 20:57:22.234618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.234593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.234801 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.234782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.245292 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.245255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdsc\" (UniqueName: \"kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.737743 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.737702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:22.740476 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:22.740446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:23.035256 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:23.035157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:23.314585 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:23.314557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:23.315626 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:57:23.315592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3e06a6_be77_43a9_8152_3192b2f584d9.slice/crio-8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755 WatchSource:0}: Error finding container 8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755: Status 404 returned error can't find the container with id 8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755
Sep 09 20:57:23.891570 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:23.891539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerStarted","Data":"428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"}
Sep 09 20:57:23.891803 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:23.891635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerStarted","Data":"8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755"}
Sep 09 20:57:25.841830 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:25.841790    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:25.901842 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:25.901815    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/0.log"
Sep 09 20:57:25.902004 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:25.901854    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerID="428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921" exitCode=1
Sep 09 20:57:25.902004 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:25.901932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerDied","Data":"428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"}
Sep 09 20:57:26.606632 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:26.606588    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:26.607040 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:26.607012    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:26.907631 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:26.907598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/0.log"
Sep 09 20:57:26.908002 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:26.907670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerStarted","Data":"f79633dda4a0b2776fb7fe33d287c0020e2c694b4481089580712046d13079d3"}
Sep 09 20:57:29.921629 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.921600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/1.log"
Sep 09 20:57:29.922054 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.922009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/0.log"
Sep 09 20:57:29.922054 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.922041    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerID="f79633dda4a0b2776fb7fe33d287c0020e2c694b4481089580712046d13079d3" exitCode=1
Sep 09 20:57:29.922160 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.922122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerDied","Data":"f79633dda4a0b2776fb7fe33d287c0020e2c694b4481089580712046d13079d3"}
Sep 09 20:57:29.922205 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.922192    2579 scope.go:117] "RemoveContainer" containerID="428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"
Sep 09 20:57:29.922572 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.922554    2579 scope.go:117] "RemoveContainer" containerID="428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"
Sep 09 20:57:29.933071 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:29.933046    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_kserve-ci-e2e-test(6c3e06a6-be77-43a9-8152-3192b2f584d9)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9"
Sep 09 20:57:29.933181 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:57:29.933139    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_kserve-ci-e2e-test_6c3e06a6-be77-43a9-8152-3192b2f584d9_0 in pod sandbox 8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755 from index: no such id: '428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921'" containerID="428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"
Sep 09 20:57:29.933276 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:29.933186    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_kserve-ci-e2e-test_6c3e06a6-be77-43a9-8152-3192b2f584d9_0 in pod sandbox 8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755 from index: no such id: '428f8d76a6b105ddcfcb0c98a8961291d1031c0d0d04824870e312d9a53c7921'"
Sep 09 20:57:30.927873 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:30.927845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/1.log"
Sep 09 20:57:32.135725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.135690    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:32.263367 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.263344    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7_6c3e06a6-be77-43a9-8152-3192b2f584d9/storage-initializer/1.log"
Sep 09 20:57:32.266944 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.266925    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:32.321194 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config\") pod \"6c3e06a6-be77-43a9-8152-3192b2f584d9\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") "
Sep 09 20:57:32.321400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgdsc\" (UniqueName: \"kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc\") pod \"6c3e06a6-be77-43a9-8152-3192b2f584d9\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") "
Sep 09 20:57:32.321400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") pod \"6c3e06a6-be77-43a9-8152-3192b2f584d9\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") "
Sep 09 20:57:32.321400 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location\") pod \"6c3e06a6-be77-43a9-8152-3192b2f584d9\" (UID: \"6c3e06a6-be77-43a9-8152-3192b2f584d9\") "
Sep 09 20:57:32.321636 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c3e06a6-be77-43a9-8152-3192b2f584d9" (UID: "6c3e06a6-be77-43a9-8152-3192b2f584d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:57:32.321758 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.321610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c3e06a6-be77-43a9-8152-3192b2f584d9" (UID: "6c3e06a6-be77-43a9-8152-3192b2f584d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:57:32.323594 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.323570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc" (OuterVolumeSpecName: "kube-api-access-vgdsc") pod "6c3e06a6-be77-43a9-8152-3192b2f584d9" (UID: "6c3e06a6-be77-43a9-8152-3192b2f584d9"). InnerVolumeSpecName "kube-api-access-vgdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:32.323699 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.323653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c3e06a6-be77-43a9-8152-3192b2f584d9" (UID: "6c3e06a6-be77-43a9-8152-3192b2f584d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:32.422213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.422171    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c3e06a6-be77-43a9-8152-3192b2f584d9-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:32.422213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.422205    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:32.422213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.422215    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdsc\" (UniqueName: \"kubernetes.io/projected/6c3e06a6-be77-43a9-8152-3192b2f584d9-kube-api-access-vgdsc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:32.422454 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.422223    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c3e06a6-be77-43a9-8152-3192b2f584d9-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:57:32.936891 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.936858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"
Sep 09 20:57:32.937075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.936898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7" event={"ID":"6c3e06a6-be77-43a9-8152-3192b2f584d9","Type":"ContainerDied","Data":"8a7ad8c01dca56e6b72704b7367c0ce0ed2093e808c56026fa60bc695ca68755"}
Sep 09 20:57:32.937075 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:32.936980    2579 scope.go:117] "RemoveContainer" containerID="f79633dda4a0b2776fb7fe33d287c0020e2c694b4481089580712046d13079d3"
Sep 09 20:57:33.022788 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.022754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:33.030381 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.030351    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-6499fd8985-hwjm7"]
Sep 09 20:57:33.331133 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:57:33.331470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331398    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.331470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331411    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.331470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331433    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.331470 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331439    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.331618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331492    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.331618 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.331501    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" containerName="storage-initializer"
Sep 09 20:57:33.335980 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.335957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.340325 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.340284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:57:33.340488 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.340469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:57:33.340601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.340576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 09 20:57:33.358297 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.358268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:57:33.430021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.429977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspns\" (UniqueName: \"kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.430220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.430033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.430220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.430052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.430220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.430108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.430220 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.430180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531160 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspns\" (UniqueName: \"kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531379 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531646 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.531986 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.532066 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.531997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.533862 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.533846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.542710 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.542689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspns\" (UniqueName: \"kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.610850 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.610771    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3e06a6-be77-43a9-8152-3192b2f584d9" path="/var/lib/kubelet/pods/6c3e06a6-be77-43a9-8152-3192b2f584d9/volumes"
Sep 09 20:57:33.646146 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.646106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:33.922352 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.922328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:57:33.924684 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:57:33.924658    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a72cc_ce3e_4251_853f_2c39a8f87c03.slice/crio-4345e6ab6f0ac60191a2e5ad4973d1c291c140a688d8e6db010658ce9b216920 WatchSource:0}: Error finding container 4345e6ab6f0ac60191a2e5ad4973d1c291c140a688d8e6db010658ce9b216920: Status 404 returned error can't find the container with id 4345e6ab6f0ac60191a2e5ad4973d1c291c140a688d8e6db010658ce9b216920
Sep 09 20:57:33.942210 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:33.942184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerStarted","Data":"4345e6ab6f0ac60191a2e5ad4973d1c291c140a688d8e6db010658ce9b216920"}
Sep 09 20:57:34.948671 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:34.948637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerStarted","Data":"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"}
Sep 09 20:57:35.841753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:35.841712    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:35.954969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:35.954931    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerID="0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8" exitCode=0
Sep 09 20:57:35.955424 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:35.955020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerDied","Data":"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"}
Sep 09 20:57:36.607263 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.607214    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:36.607683 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.607662    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:36.962137 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.962106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerStarted","Data":"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"}
Sep 09 20:57:36.962526 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.962163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerStarted","Data":"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"}
Sep 09 20:57:36.962526 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.962255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:36.994250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:36.994197    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podStartSLOduration=2.822099571 podStartE2EDuration="3.994177061s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.17207749s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:36.992688876 +0000 UTC m=+2789.829003338" watchObservedRunningTime="2026-09-09 20:57:36.994177061 +0000 UTC m=+2789.830491524"
Sep 09 20:57:37.966601 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:37.966561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:37.967994 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:37.967967    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:57:38.970656 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:38.970613    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:57:42.605899 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:42.605862    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:42.606402 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:42.606375    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:43.975505 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:43.975472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:57:43.976121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:43.976072    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:57:45.841343 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:45.841307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:57:52.606753 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:52.606709    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:57:52.607284 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:52.607151    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:57:53.976662 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:53.976626    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:57:55.841158 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:57:55.841113    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:58:02.606057 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:02.606001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:02.606554 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:02.606438    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:03.976645 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:03.976593    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:58:05.841472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:05.841436    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:58:12.605893 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:12.605849    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:12.606389 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:12.606325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:13.976972 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:13.976932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:58:15.841014 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:15.840976    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:58:22.606290 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:22.606242    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:22.606714 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:22.606693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:23.976573 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:23.976522    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:58:25.842048 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:25.842011    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:58:31.603637 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:31.603579    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:58:31.604063 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:31.603888    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" containerID="cri-o://cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad" gracePeriod=30
Sep 09 20:58:31.604063 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:31.603911    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kube-rbac-proxy" containerID="cri-o://ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809" gracePeriod=30
Sep 09 20:58:32.211496 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:32.211463    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerID="ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809" exitCode=2
Sep 09 20:58:32.211708 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:32.211524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerDied","Data":"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"}
Sep 09 20:58:32.606497 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:32.606390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:32.606907 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:32.606808    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:33.976110 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:33.976050    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:58:35.835993 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:35.835952    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 09 20:58:35.841482 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:35.841446    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 09 20:58:36.951222 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:36.951197    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:58:37.078924 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.078831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config\") pod \"a6ee9206-a125-4b12-a049-a4d2c93be418\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") "
Sep 09 20:58:37.078924 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.078886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert\") pod \"a6ee9206-a125-4b12-a049-a4d2c93be418\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") "
Sep 09 20:58:37.079213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.078964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckzfc\" (UniqueName: \"kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc\") pod \"a6ee9206-a125-4b12-a049-a4d2c93be418\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") "
Sep 09 20:58:37.079213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.078993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location\") pod \"a6ee9206-a125-4b12-a049-a4d2c93be418\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") "
Sep 09 20:58:37.079213 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.079048    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls\") pod \"a6ee9206-a125-4b12-a049-a4d2c93be418\" (UID: \"a6ee9206-a125-4b12-a049-a4d2c93be418\") "
Sep 09 20:58:37.079382 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.079313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6ee9206-a125-4b12-a049-a4d2c93be418" (UID: "a6ee9206-a125-4b12-a049-a4d2c93be418"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.079443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.079373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6ee9206-a125-4b12-a049-a4d2c93be418" (UID: "a6ee9206-a125-4b12-a049-a4d2c93be418"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:37.079443 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.079385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6ee9206-a125-4b12-a049-a4d2c93be418" (UID: "a6ee9206-a125-4b12-a049-a4d2c93be418"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:37.081373 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.081350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6ee9206-a125-4b12-a049-a4d2c93be418" (UID: "a6ee9206-a125-4b12-a049-a4d2c93be418"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:37.081457 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.081382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc" (OuterVolumeSpecName: "kube-api-access-ckzfc") pod "a6ee9206-a125-4b12-a049-a4d2c93be418" (UID: "a6ee9206-a125-4b12-a049-a4d2c93be418"). InnerVolumeSpecName "kube-api-access-ckzfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:37.180067 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.180026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.180250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.180070    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6ee9206-a125-4b12-a049-a4d2c93be418-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.180250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.180106    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckzfc\" (UniqueName: \"kubernetes.io/projected/a6ee9206-a125-4b12-a049-a4d2c93be418-kube-api-access-ckzfc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.180250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.180122    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ee9206-a125-4b12-a049-a4d2c93be418-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.180250 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.180140    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ee9206-a125-4b12-a049-a4d2c93be418-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:58:37.235299 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.235266    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerID="cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad" exitCode=0
Sep 09 20:58:37.235471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.235326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"
Sep 09 20:58:37.235471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.235344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerDied","Data":"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"}
Sep 09 20:58:37.235471 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.235437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z" event={"ID":"a6ee9206-a125-4b12-a049-a4d2c93be418","Type":"ContainerDied","Data":"78f89b67a9cc25b3a9e5b746ebd83c15151d679efd42adf6ccba20808f4a5b85"}
Sep 09 20:58:37.235624 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.235478    2579 scope.go:117] "RemoveContainer" containerID="ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"
Sep 09 20:58:37.244587 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.244563    2579 scope.go:117] "RemoveContainer" containerID="cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"
Sep 09 20:58:37.252759 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.252741    2579 scope.go:117] "RemoveContainer" containerID="5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"
Sep 09 20:58:37.258628 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.258602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:58:37.261643 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.261622    2579 scope.go:117] "RemoveContainer" containerID="ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"
Sep 09 20:58:37.261997 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:58:37.261978    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809\": container with ID starting with ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809 not found: ID does not exist" containerID="ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"
Sep 09 20:58:37.262070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262008    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809"} err="failed to get container status \"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809\": rpc error: code = NotFound desc = could not find container \"ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809\": container with ID starting with ed6a41716fffe5d8b1c3c4a296522e30f1efd7f582d02b59637538d9db6bd809 not found: ID does not exist"
Sep 09 20:58:37.262070 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262030    2579 scope.go:117] "RemoveContainer" containerID="cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"
Sep 09 20:58:37.262354 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:58:37.262333    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad\": container with ID starting with cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad not found: ID does not exist" containerID="cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"
Sep 09 20:58:37.262422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262363    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad"} err="failed to get container status \"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad\": rpc error: code = NotFound desc = could not find container \"cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad\": container with ID starting with cc49611c43e6f4e3b53ba67605a7cc4596dd871be1ce76c13c21871a98f928ad not found: ID does not exist"
Sep 09 20:58:37.262422 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262381    2579 scope.go:117] "RemoveContainer" containerID="5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"
Sep 09 20:58:37.262539 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262520    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-hwv6z"]
Sep 09 20:58:37.262664 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:58:37.262646    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e\": container with ID starting with 5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e not found: ID does not exist" containerID="5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"
Sep 09 20:58:37.262707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.262669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e"} err="failed to get container status \"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e\": rpc error: code = NotFound desc = could not find container \"5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e\": container with ID starting with 5b4f2152c042e2e7f39c405b343f4d7d146f60426f6ac33ed0513c114e2ba23e not found: ID does not exist"
Sep 09 20:58:37.616109 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:37.616049    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" path="/var/lib/kubelet/pods/a6ee9206-a125-4b12-a049-a4d2c93be418/volumes"
Sep 09 20:58:42.606666 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:42.606617    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:42.607124 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:42.607079    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:43.976188 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:43.976149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:58:52.606407 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:52.606365    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:58:52.606969 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:52.606945    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:53.977247 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:58:53.977213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:59:02.606023 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:02.605914    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:02.606491 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:02.606420    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:03.255120 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:03.254715    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:59:03.255321 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:03.255195    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" containerID="cri-o://9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb" gracePeriod=30
Sep 09 20:59:03.259121 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:03.255401    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kube-rbac-proxy" containerID="cri-o://fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302" gracePeriod=30
Sep 09 20:59:03.971416 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:03.971367    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 09 20:59:03.976811 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:03.976778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 09 20:59:04.347707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:04.347610    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerID="fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302" exitCode=2
Sep 09 20:59:04.347707 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:04.347678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerDied","Data":"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"}
Sep 09 20:59:05.606973 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:05.606930    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:05.607472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:05.607254    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:08.299300 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.299274    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:59:08.367384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.367290    2579 generic.go:356] "Generic (PLEG): container finished" podID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerID="9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb" exitCode=0
Sep 09 20:59:08.367384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.367335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerDied","Data":"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"}
Sep 09 20:59:08.367384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.367367    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"
Sep 09 20:59:08.367685 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.367424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w" event={"ID":"2c3a72cc-ce3e-4251-853f-2c39a8f87c03","Type":"ContainerDied","Data":"4345e6ab6f0ac60191a2e5ad4973d1c291c140a688d8e6db010658ce9b216920"}
Sep 09 20:59:08.367685 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.367458    2579 scope.go:117] "RemoveContainer" containerID="fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"
Sep 09 20:59:08.376900 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.376881    2579 scope.go:117] "RemoveContainer" containerID="9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"
Sep 09 20:59:08.384863 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.384843    2579 scope.go:117] "RemoveContainer" containerID="0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"
Sep 09 20:59:08.392461 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.392441    2579 scope.go:117] "RemoveContainer" containerID="fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"
Sep 09 20:59:08.392738 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:59:08.392717    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302\": container with ID starting with fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302 not found: ID does not exist" containerID="fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"
Sep 09 20:59:08.392818 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.392754    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302"} err="failed to get container status \"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302\": rpc error: code = NotFound desc = could not find container \"fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302\": container with ID starting with fb02e4f6ee7691039ac0919ba1032626236698491cb27d08894910e7c4e94302 not found: ID does not exist"
Sep 09 20:59:08.392818 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.392780    2579 scope.go:117] "RemoveContainer" containerID="9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"
Sep 09 20:59:08.393041 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:59:08.393023    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb\": container with ID starting with 9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb not found: ID does not exist" containerID="9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"
Sep 09 20:59:08.393100 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.393046    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb"} err="failed to get container status \"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb\": rpc error: code = NotFound desc = could not find container \"9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb\": container with ID starting with 9357f2bc2e2bb04cd33a6b3346df212d942f0c321ef12d11b4dc5dbe99405dfb not found: ID does not exist"
Sep 09 20:59:08.393100 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.393061    2579 scope.go:117] "RemoveContainer" containerID="0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"
Sep 09 20:59:08.393338 ip-10-0-129-45 kubenswrapper[2579]: E0909 20:59:08.393319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8\": container with ID starting with 0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8 not found: ID does not exist" containerID="0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"
Sep 09 20:59:08.393384 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.393343    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8"} err="failed to get container status \"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8\": rpc error: code = NotFound desc = could not find container \"0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8\": container with ID starting with 0d33eb21ff78cb90f315e19c508f0a4ff0c3b3a7f284b3d6759600d7cd344ae8 not found: ID does not exist"
Sep 09 20:59:08.453279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config\") pod \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") "
Sep 09 20:59:08.453279 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location\") pod \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") "
Sep 09 20:59:08.453493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls\") pod \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") "
Sep 09 20:59:08.453493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453411    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nspns\" (UniqueName: \"kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns\") pod \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") "
Sep 09 20:59:08.453493 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert\") pod \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\" (UID: \"2c3a72cc-ce3e-4251-853f-2c39a8f87c03\") "
Sep 09 20:59:08.453684 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c3a72cc-ce3e-4251-853f-2c39a8f87c03" (UID: "2c3a72cc-ce3e-4251-853f-2c39a8f87c03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:08.453784 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c3a72cc-ce3e-4251-853f-2c39a8f87c03" (UID: "2c3a72cc-ce3e-4251-853f-2c39a8f87c03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:08.453900 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.453865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c3a72cc-ce3e-4251-853f-2c39a8f87c03" (UID: "2c3a72cc-ce3e-4251-853f-2c39a8f87c03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:08.455630 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.455603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c3a72cc-ce3e-4251-853f-2c39a8f87c03" (UID: "2c3a72cc-ce3e-4251-853f-2c39a8f87c03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:08.455733 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.455715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns" (OuterVolumeSpecName: "kube-api-access-nspns") pod "2c3a72cc-ce3e-4251-853f-2c39a8f87c03" (UID: "2c3a72cc-ce3e-4251-853f-2c39a8f87c03"). InnerVolumeSpecName "kube-api-access-nspns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:08.554451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.554398    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspns\" (UniqueName: \"kubernetes.io/projected/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-api-access-nspns\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:59:08.554451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.554448    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:59:08.554451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.554458    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:59:08.554451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.554467    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:59:08.554731 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.554476    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c3a72cc-ce3e-4251-853f-2c39a8f87c03-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 20:59:08.691734 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.691696    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:59:08.694341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:08.694313    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-85d7777d97-lmd9w"]
Sep 09 20:59:09.611354 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:09.611319    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" path="/var/lib/kubelet/pods/2c3a72cc-ce3e-4251-853f-2c39a8f87c03/volumes"
Sep 09 20:59:15.579746 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.579710    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580020    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580029    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580035    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kube-rbac-proxy"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580048    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kube-rbac-proxy"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580055    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kube-rbac-proxy"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580062    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kube-rbac-proxy"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580074    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="storage-initializer"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580080    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="storage-initializer"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580111    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="storage-initializer"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="storage-initializer"
Sep 09 20:59:15.580169 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580168    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kserve-container"
Sep 09 20:59:15.580607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580181    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kserve-container"
Sep 09 20:59:15.580607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580186    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ee9206-a125-4b12-a049-a4d2c93be418" containerName="kube-rbac-proxy"
Sep 09 20:59:15.580607 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.580192    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3a72cc-ce3e-4251-853f-2c39a8f87c03" containerName="kube-rbac-proxy"
Sep 09 20:59:15.584983 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.584962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.587835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.587810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:59:15.587835 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.587823    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 09 20:59:15.588008 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.587825    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:59:15.594769 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.594744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 20:59:15.605589 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.605552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.605731 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.605621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v426m\" (UniqueName: \"kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.605731 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.605691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.605823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.605767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.605823 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.605809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.607402 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.607370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:15.607947 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.607926    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:15.706819 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.706782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.706832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v426m\" (UniqueName: \"kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.706893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.706940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707021 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.706974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707500 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.707475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707711 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.707691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.707765 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.707753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.709540 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.709521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.716846 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.716824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v426m\" (UniqueName: \"kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:15.895760 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:15.895726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:16.169032 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:16.169008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 20:59:16.171500 ip-10-0-129-45 kubenswrapper[2579]: W0909 20:59:16.171468    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6f6238_43d9_42b9_a8c6_5857acc6a7a6.slice/crio-780e55e1c6081b826ce3c8f5b16a93a16f1e34134042ff161d5af3eb944f9347 WatchSource:0}: Error finding container 780e55e1c6081b826ce3c8f5b16a93a16f1e34134042ff161d5af3eb944f9347: Status 404 returned error can't find the container with id 780e55e1c6081b826ce3c8f5b16a93a16f1e34134042ff161d5af3eb944f9347
Sep 09 20:59:16.407018 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:16.406980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerStarted","Data":"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"}
Sep 09 20:59:16.407258 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:16.407043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerStarted","Data":"780e55e1c6081b826ce3c8f5b16a93a16f1e34134042ff161d5af3eb944f9347"}
Sep 09 20:59:18.415867 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:18.415834    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerID="e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f" exitCode=0
Sep 09 20:59:18.416282 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:18.415896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerDied","Data":"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"}
Sep 09 20:59:19.423431 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:19.423392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerStarted","Data":"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"}
Sep 09 20:59:19.423824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:19.423486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerStarted","Data":"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"}
Sep 09 20:59:19.423824 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:19.423750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:19.444780 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:19.444722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podStartSLOduration=3.279256436 podStartE2EDuration="4.444707699s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.165451263s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:19.443523535 +0000 UTC m=+2892.279838039" watchObservedRunningTime="2026-09-09 20:59:19.444707699 +0000 UTC m=+2892.281022161"
Sep 09 20:59:20.427451 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:20.427416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:20.428625 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:20.428597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 20:59:21.431363 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:21.431316    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 20:59:25.606911 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:25.606859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:25.607341 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:25.607311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:26.435465 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:26.435434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 20:59:26.435929 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:26.435902    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 20:59:35.607177 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:35.607132    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:35.607725 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:35.607691    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:36.435921 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:36.435878    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 20:59:45.606931 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:45.606887    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:45.607472 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:45.607411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:46.436229 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:46.436188    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 20:59:55.607027 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:55.606978    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:59:55.607486 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:55.607465    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:59:56.436657 ip-10-0-129-45 kubenswrapper[2579]: I0909 20:59:56.436611    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 21:00:05.606957 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:05.606899    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:05.607453 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:05.607419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:06.436269 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:06.436226    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 21:00:15.606850 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:15.606802    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:15.607360 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:15.607325    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:16.436642 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:16.436597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 21:00:18.606265 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:18.606221    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:18.606749 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:18.606581    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:26.436658 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:26.436610    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 21:00:28.606564 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:28.606463    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:28.607031 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:28.606996    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:36.437148 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:36.437114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 21:00:38.606615 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:38.606572    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:38.607060 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:38.606939    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:45.506472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:45.506432    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 21:00:45.506887 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:45.506726    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" containerID="cri-o://8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941" gracePeriod=30
Sep 09 21:00:45.506887 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:45.506726    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kube-rbac-proxy" containerID="cri-o://0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf" gracePeriod=30
Sep 09 21:00:45.783070 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:45.782980    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerID="0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf" exitCode=2
Sep 09 21:00:45.783070 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:45.783047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerDied","Data":"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"}
Sep 09 21:00:46.432309 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:46.432263    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Sep 09 21:00:46.436585 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:46.436550    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 09 21:00:48.606227 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:48.606166    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:48.606718 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:48.606691    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:50.653924 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.653898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 21:00:50.805539 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config\") pod \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") "
Sep 09 21:00:50.805539 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v426m\" (UniqueName: \"kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m\") pod \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") "
Sep 09 21:00:50.805539 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location\") pod \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") "
Sep 09 21:00:50.805830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls\") pod \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") "
Sep 09 21:00:50.805830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert\") pod \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\" (UID: \"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6\") "
Sep 09 21:00:50.805950 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" (UID: "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:50.806015 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.805978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" (UID: "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:50.806190 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.806156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" (UID: "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:50.808054 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808022    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerID="8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941" exitCode=0
Sep 09 21:00:50.808187 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerDied","Data":"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"}
Sep 09 21:00:50.808187 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"
Sep 09 21:00:50.808287 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94" event={"ID":"8f6f6238-43d9-42b9-a8c6-5857acc6a7a6","Type":"ContainerDied","Data":"780e55e1c6081b826ce3c8f5b16a93a16f1e34134042ff161d5af3eb944f9347"}
Sep 09 21:00:50.808287 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m" (OuterVolumeSpecName: "kube-api-access-v426m") pod "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" (UID: "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6"). InnerVolumeSpecName "kube-api-access-v426m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:50.808287 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" (UID: "8f6f6238-43d9-42b9-a8c6-5857acc6a7a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:50.808287 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.808226    2579 scope.go:117] "RemoveContainer" containerID="0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"
Sep 09 21:00:50.826078 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.826061    2579 scope.go:117] "RemoveContainer" containerID="8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"
Sep 09 21:00:50.833684 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.833665    2579 scope.go:117] "RemoveContainer" containerID="e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"
Sep 09 21:00:50.838555 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.838533    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 21:00:50.841690 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.841669    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-78cfc98658-2js94"]
Sep 09 21:00:50.842497 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.842475    2579 scope.go:117] "RemoveContainer" containerID="0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"
Sep 09 21:00:50.842823 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:00:50.842805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf\": container with ID starting with 0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf not found: ID does not exist" containerID="0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"
Sep 09 21:00:50.842885 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.842832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf"} err="failed to get container status \"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf\": rpc error: code = NotFound desc = could not find container \"0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf\": container with ID starting with 0e2308b59e65da103eb4762e19b4add2806b63c07de079991161ac4ed5c47fcf not found: ID does not exist"
Sep 09 21:00:50.842885 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.842852    2579 scope.go:117] "RemoveContainer" containerID="8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"
Sep 09 21:00:50.843132 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:00:50.843107    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941\": container with ID starting with 8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941 not found: ID does not exist" containerID="8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"
Sep 09 21:00:50.843234 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.843139    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941"} err="failed to get container status \"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941\": rpc error: code = NotFound desc = could not find container \"8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941\": container with ID starting with 8828a3c3b5a6c3b9db736825b6a4de141a5cb75106bfb1a085060f72e8177941 not found: ID does not exist"
Sep 09 21:00:50.843234 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.843165    2579 scope.go:117] "RemoveContainer" containerID="e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"
Sep 09 21:00:50.843426 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:00:50.843397    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f\": container with ID starting with e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f not found: ID does not exist" containerID="e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"
Sep 09 21:00:50.843502 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.843433    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f"} err="failed to get container status \"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f\": rpc error: code = NotFound desc = could not find container \"e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f\": container with ID starting with e5894356c278e7f9774d11e3e1d275c20cbda8017098c75e3e43b0622e09645f not found: ID does not exist"
Sep 09 21:00:50.906756 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.906716    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.906756 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.906751    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.906756 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.906762    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v426m\" (UniqueName: \"kubernetes.io/projected/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kube-api-access-v426m\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.906990 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.906774    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.906990 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:50.906782    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:00:51.610523 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:51.610480    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" path="/var/lib/kubelet/pods/8f6f6238-43d9-42b9-a8c6-5857acc6a7a6/volumes"
Sep 09 21:00:57.734049 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734360    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="storage-initializer"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734374    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="storage-initializer"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734388    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kube-rbac-proxy"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734408    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kube-rbac-proxy"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734463    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kube-rbac-proxy"
Sep 09 21:00:57.734472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.734473    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6f6238-43d9-42b9-a8c6-5857acc6a7a6" containerName="kserve-container"
Sep 09 21:00:57.739010 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.738993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.741839 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.741811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 09 21:00:57.741990 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.741855    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:00:57.747071 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.747046    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:00:57.865322 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.865282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bln\" (UniqueName: \"kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.865322 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.865319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.865544 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.865360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.865544 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.865382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.865544 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.865422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.966741 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.966703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bln\" (UniqueName: \"kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.966741 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.966743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.966997 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.966780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.966997 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.966805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.966997 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.966860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.967275 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.967253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.967546 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.967529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.967605 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.967535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.969507 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.969489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:57.975931 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:57.975908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bln\" (UniqueName: \"kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln\") pod \"isvc-paddle-runtime-predictor-58b99fff76-wsrbd\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:58.050889 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.050775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:00:58.323057 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.323032    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:00:58.324834 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:00:58.324808    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode421a568_650d_462b_967f_386eea8d32e7.slice/crio-3b5682b5fb69d2fcc3ab1a01ee56efaf762a4c2ac4ada8cd70f2889e21c8a7de WatchSource:0}: Error finding container 3b5682b5fb69d2fcc3ab1a01ee56efaf762a4c2ac4ada8cd70f2889e21c8a7de: Status 404 returned error can't find the container with id 3b5682b5fb69d2fcc3ab1a01ee56efaf762a4c2ac4ada8cd70f2889e21c8a7de
Sep 09 21:00:58.607199 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.607069    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:00:58.607555 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.607525    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:00:58.843807 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.843775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerStarted","Data":"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"}
Sep 09 21:00:58.844197 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:00:58.843837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerStarted","Data":"3b5682b5fb69d2fcc3ab1a01ee56efaf762a4c2ac4ada8cd70f2889e21c8a7de"}
Sep 09 21:01:03.866343 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:03.866242    2579 generic.go:356] "Generic (PLEG): container finished" podID="e421a568-650d-462b-967f-386eea8d32e7" containerID="408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8" exitCode=0
Sep 09 21:01:03.866343 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:03.866326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerDied","Data":"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"}
Sep 09 21:01:03.867758 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:03.867740    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:01:08.606463 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:08.606406    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:08.606976 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:08.606942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:15.931132 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:15.931061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerStarted","Data":"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"}
Sep 09 21:01:15.931570 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:15.931161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerStarted","Data":"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"}
Sep 09 21:01:15.931570 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:15.931365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:01:15.950515 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:15.950464    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podStartSLOduration=1.983444743 podStartE2EDuration="18.950447815s" totalImagesPullingTime="11.84879562s" totalInitContainerRuntime="5.118207452s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:15.949285467 +0000 UTC m=+3008.785599943" watchObservedRunningTime="2026-09-09 21:01:15.950447815 +0000 UTC m=+3008.786762279"
Sep 09 21:01:16.936257 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:16.936221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:01:16.937592 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:16.937555    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:17.940639 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:17.940594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:18.606317 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:18.606270    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:18.606702 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:18.606672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:22.944992 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:22.944960    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:01:22.945511 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:22.945483    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:28.606433 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:28.606381    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:28.606900 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:28.606873    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:32.946006 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:32.945966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:38.606320 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:38.606268    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:38.606781 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:38.606757    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:40.606427 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:40.606383    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:40.606906 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:40.606767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:42.946286 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:42.946243    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:50.606409 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:50.606353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:01:50.606905 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:50.606870    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:01:52.945872 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:52.945826    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 09 21:01:53.244849 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.244770    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:01:53.247370 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.247350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.249896 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.249872    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 09 21:01:53.250021 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.250002    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:01:53.258955 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.258927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:01:53.270712 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.270682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.270842 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.270723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcn2\" (UniqueName: \"kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.270842 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.270746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.270924 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.270855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.270924 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.270875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371457 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371457 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371743 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371743 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcn2\" (UniqueName: \"kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371743 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.371743 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:01:53.371683    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 09 21:01:53.371959 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:01:53.371792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls podName:70afefe5-3f68-45a3-a952-5907bfe5d549 nodeName:}" failed. No retries permitted until 2026-09-09 21:01:53.871750228 +0000 UTC m=+3046.708064674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 09 21:01:53.371959 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.371866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.372177 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.372157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.372228 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.372213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.380471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.380446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcn2\" (UniqueName: \"kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.875728 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.875686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:53.878462 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:53.878431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:54.157196 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:54.157168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:01:54.428953 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:54.428927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:01:54.430948 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:01:54.430920    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70afefe5_3f68_45a3_a952_5907bfe5d549.slice/crio-83a0bff79d65b0d6768a7c98e3b1b8fab0e76cdf3106dff7292b2cdeb7f4ff07 WatchSource:0}: Error finding container 83a0bff79d65b0d6768a7c98e3b1b8fab0e76cdf3106dff7292b2cdeb7f4ff07: Status 404 returned error can't find the container with id 83a0bff79d65b0d6768a7c98e3b1b8fab0e76cdf3106dff7292b2cdeb7f4ff07
Sep 09 21:01:55.097278 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:55.097244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerStarted","Data":"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"}
Sep 09 21:01:55.097456 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:55.097309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerStarted","Data":"83a0bff79d65b0d6768a7c98e3b1b8fab0e76cdf3106dff7292b2cdeb7f4ff07"}
Sep 09 21:01:59.116970 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:59.116930    2579 generic.go:356] "Generic (PLEG): container finished" podID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerID="28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3" exitCode=0
Sep 09 21:01:59.117500 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:01:59.116988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerDied","Data":"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"}
Sep 09 21:02:00.124115 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.124053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerStarted","Data":"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"}
Sep 09 21:02:00.124600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.124178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerStarted","Data":"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"}
Sep 09 21:02:00.124600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.124418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:00.145706 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.145651    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" podStartSLOduration=3.234860908 podStartE2EDuration="7.145631093s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.910770185s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:00.143709278 +0000 UTC m=+3052.980023744" watchObservedRunningTime="2026-09-09 21:02:00.145631093 +0000 UTC m=+3052.981945560"
Sep 09 21:02:00.606533 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.606477    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:00.606929 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:00.606891    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:01.128728 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:01.128696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:02.946236 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:02.946208    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:02:07.140308 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:07.140280    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:08.820840 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:08.820804    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:02:08.821268 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:08.821050    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container" containerID="cri-o://bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98" gracePeriod=30
Sep 09 21:02:08.821268 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:08.821131    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kube-rbac-proxy" containerID="cri-o://059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498" gracePeriod=30
Sep 09 21:02:09.165028 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:09.164990    2579 generic.go:356] "Generic (PLEG): container finished" podID="e421a568-650d-462b-967f-386eea8d32e7" containerID="059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498" exitCode=2
Sep 09 21:02:09.167263 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:09.165063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerDied","Data":"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"}
Sep 09 21:02:10.606451 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:10.606402    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:10.606886 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:10.606861    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:11.663600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.663568    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:02:11.735339 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls\") pod \"e421a568-650d-462b-967f-386eea8d32e7\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") "
Sep 09 21:02:11.735532 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config\") pod \"e421a568-650d-462b-967f-386eea8d32e7\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") "
Sep 09 21:02:11.735532 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7bln\" (UniqueName: \"kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln\") pod \"e421a568-650d-462b-967f-386eea8d32e7\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") "
Sep 09 21:02:11.735532 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert\") pod \"e421a568-650d-462b-967f-386eea8d32e7\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") "
Sep 09 21:02:11.735722 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location\") pod \"e421a568-650d-462b-967f-386eea8d32e7\" (UID: \"e421a568-650d-462b-967f-386eea8d32e7\") "
Sep 09 21:02:11.735946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e421a568-650d-462b-967f-386eea8d32e7" (UID: "e421a568-650d-462b-967f-386eea8d32e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:11.736110 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.735993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e421a568-650d-462b-967f-386eea8d32e7" (UID: "e421a568-650d-462b-967f-386eea8d32e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:11.737716 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.737689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e421a568-650d-462b-967f-386eea8d32e7" (UID: "e421a568-650d-462b-967f-386eea8d32e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:11.737837 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.737815    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln" (OuterVolumeSpecName: "kube-api-access-k7bln") pod "e421a568-650d-462b-967f-386eea8d32e7" (UID: "e421a568-650d-462b-967f-386eea8d32e7"). InnerVolumeSpecName "kube-api-access-k7bln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:11.750496 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.750432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e421a568-650d-462b-967f-386eea8d32e7" (UID: "e421a568-650d-462b-967f-386eea8d32e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:11.836644 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.836608    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bln\" (UniqueName: \"kubernetes.io/projected/e421a568-650d-462b-967f-386eea8d32e7-kube-api-access-k7bln\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.836644 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.836640    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.836644 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.836649    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e421a568-650d-462b-967f-386eea8d32e7-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.836888 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.836659    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e421a568-650d-462b-967f-386eea8d32e7-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.836888 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:11.836668    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e421a568-650d-462b-967f-386eea8d32e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:12.178862 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.178826    2579 generic.go:356] "Generic (PLEG): container finished" podID="e421a568-650d-462b-967f-386eea8d32e7" containerID="bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98" exitCode=0
Sep 09 21:02:12.179050 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.178902    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"
Sep 09 21:02:12.179050 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.178903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerDied","Data":"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"}
Sep 09 21:02:12.179050 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.178987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd" event={"ID":"e421a568-650d-462b-967f-386eea8d32e7","Type":"ContainerDied","Data":"3b5682b5fb69d2fcc3ab1a01ee56efaf762a4c2ac4ada8cd70f2889e21c8a7de"}
Sep 09 21:02:12.179050 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.179015    2579 scope.go:117] "RemoveContainer" containerID="059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"
Sep 09 21:02:12.187911 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.187892    2579 scope.go:117] "RemoveContainer" containerID="bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"
Sep 09 21:02:12.195851 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.195826    2579 scope.go:117] "RemoveContainer" containerID="408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"
Sep 09 21:02:12.201556 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.201531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:02:12.204381 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.204355    2579 scope.go:117] "RemoveContainer" containerID="059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"
Sep 09 21:02:12.204721 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:12.204701    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498\": container with ID starting with 059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498 not found: ID does not exist" containerID="059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"
Sep 09 21:02:12.204790 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.204732    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498"} err="failed to get container status \"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498\": rpc error: code = NotFound desc = could not find container \"059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498\": container with ID starting with 059f563a0a76403c2c6f1938c5a12e104d1d710614a8f7b6b68b2e7043884498 not found: ID does not exist"
Sep 09 21:02:12.204790 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.204752    2579 scope.go:117] "RemoveContainer" containerID="bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"
Sep 09 21:02:12.205034 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:12.205018    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98\": container with ID starting with bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98 not found: ID does not exist" containerID="bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"
Sep 09 21:02:12.205108 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.205038    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98"} err="failed to get container status \"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98\": rpc error: code = NotFound desc = could not find container \"bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98\": container with ID starting with bc9522d28d1c9431438c74661e8cd4471600be50908a4d5a5589f989dd854f98 not found: ID does not exist"
Sep 09 21:02:12.205108 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.205051    2579 scope.go:117] "RemoveContainer" containerID="408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"
Sep 09 21:02:12.205377 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:12.205358    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8\": container with ID starting with 408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8 not found: ID does not exist" containerID="408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"
Sep 09 21:02:12.205433 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.205382    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8"} err="failed to get container status \"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8\": rpc error: code = NotFound desc = could not find container \"408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8\": container with ID starting with 408cb7e7b726b0ee3b4b02eef448543440a31ba09e32ed3b01f4e7dbb41d5ca8 not found: ID does not exist"
Sep 09 21:02:12.205986 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:12.205970    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-wsrbd"]
Sep 09 21:02:13.611243 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:13.611207    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e421a568-650d-462b-967f-386eea8d32e7" path="/var/lib/kubelet/pods/e421a568-650d-462b-967f-386eea8d32e7/volumes"
Sep 09 21:02:20.606353 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:20.606298    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:20.606802 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:20.606778    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:30.606859 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:30.606809    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:30.607384 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:30.607288    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:37.144272 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:37.144239    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:40.606403 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:40.606351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:40.606844 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:40.606806    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:43.391538 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:43.391484    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:02:43.392041 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:43.391755    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kserve-container" containerID="cri-o://ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee" gracePeriod=30
Sep 09 21:02:43.392041 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:43.391774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kube-rbac-proxy" containerID="cri-o://5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6" gracePeriod=30
Sep 09 21:02:44.312855 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.312812    2579 generic.go:356] "Generic (PLEG): container finished" podID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerID="5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6" exitCode=2
Sep 09 21:02:44.313045 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.312887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerDied","Data":"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"}
Sep 09 21:02:44.530377 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.530349    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:44.619840 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.619749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location\") pod \"70afefe5-3f68-45a3-a952-5907bfe5d549\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") "
Sep 09 21:02:44.619840 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.619790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwcn2\" (UniqueName: \"kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2\") pod \"70afefe5-3f68-45a3-a952-5907bfe5d549\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") "
Sep 09 21:02:44.619840 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.619830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config\") pod \"70afefe5-3f68-45a3-a952-5907bfe5d549\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") "
Sep 09 21:02:44.619840 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.619844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert\") pod \"70afefe5-3f68-45a3-a952-5907bfe5d549\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") "
Sep 09 21:02:44.620222 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.619872    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") pod \"70afefe5-3f68-45a3-a952-5907bfe5d549\" (UID: \"70afefe5-3f68-45a3-a952-5907bfe5d549\") "
Sep 09 21:02:44.620222 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.620172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70afefe5-3f68-45a3-a952-5907bfe5d549" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:44.620222 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.620190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70afefe5-3f68-45a3-a952-5907bfe5d549" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:44.620222 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.620202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70afefe5-3f68-45a3-a952-5907bfe5d549" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:44.622147 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.622112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70afefe5-3f68-45a3-a952-5907bfe5d549" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:44.622279 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.622170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2" (OuterVolumeSpecName: "kube-api-access-wwcn2") pod "70afefe5-3f68-45a3-a952-5907bfe5d549" (UID: "70afefe5-3f68-45a3-a952-5907bfe5d549"). InnerVolumeSpecName "kube-api-access-wwcn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:44.720744 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.720700    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70afefe5-3f68-45a3-a952-5907bfe5d549-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:44.720744 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.720739    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcn2\" (UniqueName: \"kubernetes.io/projected/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-api-access-wwcn2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:44.720744 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.720750    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:44.721019 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.720759    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70afefe5-3f68-45a3-a952-5907bfe5d549-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:44.721019 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:44.720769    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70afefe5-3f68-45a3-a952-5907bfe5d549-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:02:45.320650 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.320617    2579 generic.go:356] "Generic (PLEG): container finished" podID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerID="ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee" exitCode=0
Sep 09 21:02:45.320871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.320684    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"
Sep 09 21:02:45.320871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.320703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerDied","Data":"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"}
Sep 09 21:02:45.320871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.320768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx" event={"ID":"70afefe5-3f68-45a3-a952-5907bfe5d549","Type":"ContainerDied","Data":"83a0bff79d65b0d6768a7c98e3b1b8fab0e76cdf3106dff7292b2cdeb7f4ff07"}
Sep 09 21:02:45.320871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.320796    2579 scope.go:117] "RemoveContainer" containerID="5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"
Sep 09 21:02:45.331753 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.330970    2579 scope.go:117] "RemoveContainer" containerID="ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"
Sep 09 21:02:45.339896 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.339875    2579 scope.go:117] "RemoveContainer" containerID="28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"
Sep 09 21:02:45.345273 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.345246    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:02:45.348349 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.348308    2579 scope.go:117] "RemoveContainer" containerID="5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"
Sep 09 21:02:45.348864 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:45.348838    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6\": container with ID starting with 5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6 not found: ID does not exist" containerID="5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"
Sep 09 21:02:45.348951 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.348872    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6"} err="failed to get container status \"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6\": rpc error: code = NotFound desc = could not find container \"5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6\": container with ID starting with 5e0722536a5512b07a1b9aee07e58dd13a9c45ef366ef97a992800209bf1f4b6 not found: ID does not exist"
Sep 09 21:02:45.348951 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.348895    2579 scope.go:117] "RemoveContainer" containerID="ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"
Sep 09 21:02:45.349242 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:45.349220    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee\": container with ID starting with ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee not found: ID does not exist" containerID="ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"
Sep 09 21:02:45.349304 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.349247    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee"} err="failed to get container status \"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee\": rpc error: code = NotFound desc = could not find container \"ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee\": container with ID starting with ac810244bda43510a0e856e1819ded779e4b91d01b898cd91c7307670a1b52ee not found: ID does not exist"
Sep 09 21:02:45.349304 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.349262    2579 scope.go:117] "RemoveContainer" containerID="28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"
Sep 09 21:02:45.349536 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:02:45.349521    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3\": container with ID starting with 28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3 not found: ID does not exist" containerID="28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"
Sep 09 21:02:45.349579 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.349539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3"} err="failed to get container status \"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3\": rpc error: code = NotFound desc = could not find container \"28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3\": container with ID starting with 28be820fa19c5b5a09a8bd7ce95f45ac72658f2dadb18f99d8e3bfff628454f3 not found: ID does not exist"
Sep 09 21:02:45.350284 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.350264    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2stqx"]
Sep 09 21:02:45.617443 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:45.617365    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" path="/var/lib/kubelet/pods/70afefe5-3f68-45a3-a952-5907bfe5d549/volumes"
Sep 09 21:02:48.580653 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580935    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580946    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="storage-initializer"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580952    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="storage-initializer"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580958    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kserve-container"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580964    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kserve-container"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580973    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580979    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580988    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.580993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581004    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="storage-initializer"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581010    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="storage-initializer"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581051    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581058    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kserve-container"
Sep 09 21:02:48.581058 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581065    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e421a568-650d-462b-967f-386eea8d32e7" containerName="kube-rbac-proxy"
Sep 09 21:02:48.581611 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.581073    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70afefe5-3f68-45a3-a952-5907bfe5d549" containerName="kserve-container"
Sep 09 21:02:48.585633 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.585613    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.588321 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.588298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:02:48.588459 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.588297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 09 21:02:48.593599 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.593575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:02:48.654224 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.654186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rw2\" (UniqueName: \"kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.654449 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.654245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.654449 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.654287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.654449 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.654352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.654449 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.654381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755022 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.754988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rw2\" (UniqueName: \"kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755250 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755250 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755250 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755250 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755621 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755868 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.755927 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.755870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.757761 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.757744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.763801 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.763781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rw2\" (UniqueName: \"kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:48.897003 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:48.896961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:02:49.173750 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:49.173726    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:02:49.175783 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:02:49.175754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f58c1d_c595_415f_8809_adc720bece8d.slice/crio-bbe75a6f912cf40eb6acf215e08372a85855f4ea59a077c4c584e58b0d4fca9f WatchSource:0}: Error finding container bbe75a6f912cf40eb6acf215e08372a85855f4ea59a077c4c584e58b0d4fca9f: Status 404 returned error can't find the container with id bbe75a6f912cf40eb6acf215e08372a85855f4ea59a077c4c584e58b0d4fca9f
Sep 09 21:02:49.340740 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:49.340699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerStarted","Data":"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"}
Sep 09 21:02:49.340925 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:49.340768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerStarted","Data":"bbe75a6f912cf40eb6acf215e08372a85855f4ea59a077c4c584e58b0d4fca9f"}
Sep 09 21:02:50.606534 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:50.606488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:02:50.606958 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:50.606903    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:53.357229 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:53.357193    2579 generic.go:356] "Generic (PLEG): container finished" podID="f0f58c1d-c595-415f-8809-adc720bece8d" containerID="51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323" exitCode=0
Sep 09 21:02:53.357629 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:02:53.357267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerDied","Data":"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"}
Sep 09 21:03:00.395963 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.395924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerStarted","Data":"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"}
Sep 09 21:03:00.396489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.396037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerStarted","Data":"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"}
Sep 09 21:03:00.396489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.396424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:00.396608 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.396559    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:00.397914 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.397887    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 09 21:03:00.417062 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.417011    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podStartSLOduration=1.8487204149999998 podStartE2EDuration="12.416995419s" totalImagesPullingTime="6.479330764s" totalInitContainerRuntime="4.08894424s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:00.41613722 +0000 UTC m=+3113.252451684" watchObservedRunningTime="2026-09-09 21:03:00.416995419 +0000 UTC m=+3113.253309882"
Sep 09 21:03:00.606509 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.606457    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:00.606798 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.606765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:00.607033 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.607007    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:00.607704 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:00.607678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:01.400563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:01.400516    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 09 21:03:06.405176 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:06.405141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:06.405783 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:06.405615    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Sep 09 21:03:10.606860 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:10.606818    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:10.608517 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:10.608495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:16.407184 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:16.407151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:18.630328 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:18.630286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:03:18.630824 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:18.630561    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container" containerID="cri-o://c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d" gracePeriod=30
Sep 09 21:03:18.630824 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:18.630620    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kube-rbac-proxy" containerID="cri-o://54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c" gracePeriod=30
Sep 09 21:03:19.376163 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.376134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:19.482145 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482040    2579 generic.go:356] "Generic (PLEG): container finished" podID="f0f58c1d-c595-415f-8809-adc720bece8d" containerID="54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c" exitCode=2
Sep 09 21:03:19.482145 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482066    2579 generic.go:356] "Generic (PLEG): container finished" podID="f0f58c1d-c595-415f-8809-adc720bece8d" containerID="c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d" exitCode=0
Sep 09 21:03:19.482145 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"
Sep 09 21:03:19.482410 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerDied","Data":"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"}
Sep 09 21:03:19.482410 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerDied","Data":"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"}
Sep 09 21:03:19.482410 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6" event={"ID":"f0f58c1d-c595-415f-8809-adc720bece8d","Type":"ContainerDied","Data":"bbe75a6f912cf40eb6acf215e08372a85855f4ea59a077c4c584e58b0d4fca9f"}
Sep 09 21:03:19.482410 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.482294    2579 scope.go:117] "RemoveContainer" containerID="54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"
Sep 09 21:03:19.490575 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.490559    2579 scope.go:117] "RemoveContainer" containerID="c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"
Sep 09 21:03:19.498003 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.497983    2579 scope.go:117] "RemoveContainer" containerID="51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"
Sep 09 21:03:19.505315 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.505296    2579 scope.go:117] "RemoveContainer" containerID="54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"
Sep 09 21:03:19.505625 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:03:19.505603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c\": container with ID starting with 54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c not found: ID does not exist" containerID="54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"
Sep 09 21:03:19.505676 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.505636    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"} err="failed to get container status \"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c\": rpc error: code = NotFound desc = could not find container \"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c\": container with ID starting with 54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c not found: ID does not exist"
Sep 09 21:03:19.505676 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.505656    2579 scope.go:117] "RemoveContainer" containerID="c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"
Sep 09 21:03:19.505929 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:03:19.505912    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d\": container with ID starting with c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d not found: ID does not exist" containerID="c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"
Sep 09 21:03:19.505994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.505941    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"} err="failed to get container status \"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d\": rpc error: code = NotFound desc = could not find container \"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d\": container with ID starting with c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d not found: ID does not exist"
Sep 09 21:03:19.505994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.505967    2579 scope.go:117] "RemoveContainer" containerID="51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"
Sep 09 21:03:19.506251 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:03:19.506231    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323\": container with ID starting with 51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323 not found: ID does not exist" containerID="51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"
Sep 09 21:03:19.506290 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506258    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"} err="failed to get container status \"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323\": rpc error: code = NotFound desc = could not find container \"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323\": container with ID starting with 51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323 not found: ID does not exist"
Sep 09 21:03:19.506290 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506273    2579 scope.go:117] "RemoveContainer" containerID="54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"
Sep 09 21:03:19.506498 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506479    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c"} err="failed to get container status \"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c\": rpc error: code = NotFound desc = could not find container \"54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c\": container with ID starting with 54aa3d39895ff7c53e27a381166d2f7a2e6df28ae7378aa790d3f61ccdf97a1c not found: ID does not exist"
Sep 09 21:03:19.506570 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506500    2579 scope.go:117] "RemoveContainer" containerID="c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"
Sep 09 21:03:19.506745 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506727    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d"} err="failed to get container status \"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d\": rpc error: code = NotFound desc = could not find container \"c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d\": container with ID starting with c3dfaa78bac2335adce284b4936e94595c73e68b40e7e80e9299b89c3d4b341d not found: ID does not exist"
Sep 09 21:03:19.506793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506746    2579 scope.go:117] "RemoveContainer" containerID="51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"
Sep 09 21:03:19.506956 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.506939    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323"} err="failed to get container status \"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323\": rpc error: code = NotFound desc = could not find container \"51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323\": container with ID starting with 51bcbbff82c8365c96d7d3eb6811450c61301045685877efc639b7d5b7b98323 not found: ID does not exist"
Sep 09 21:03:19.526306 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8rw2\" (UniqueName: \"kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2\") pod \"f0f58c1d-c595-415f-8809-adc720bece8d\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") "
Sep 09 21:03:19.526378 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526320    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert\") pod \"f0f58c1d-c595-415f-8809-adc720bece8d\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") "
Sep 09 21:03:19.526378 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location\") pod \"f0f58c1d-c595-415f-8809-adc720bece8d\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") "
Sep 09 21:03:19.526378 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls\") pod \"f0f58c1d-c595-415f-8809-adc720bece8d\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") "
Sep 09 21:03:19.526519 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config\") pod \"f0f58c1d-c595-415f-8809-adc720bece8d\" (UID: \"f0f58c1d-c595-415f-8809-adc720bece8d\") "
Sep 09 21:03:19.526736 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0f58c1d-c595-415f-8809-adc720bece8d" (UID: "f0f58c1d-c595-415f-8809-adc720bece8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:03:19.526828 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0f58c1d-c595-415f-8809-adc720bece8d" (UID: "f0f58c1d-c595-415f-8809-adc720bece8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:19.526828 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.526753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0f58c1d-c595-415f-8809-adc720bece8d" (UID: "f0f58c1d-c595-415f-8809-adc720bece8d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:03:19.528535 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.528516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2" (OuterVolumeSpecName: "kube-api-access-h8rw2") pod "f0f58c1d-c595-415f-8809-adc720bece8d" (UID: "f0f58c1d-c595-415f-8809-adc720bece8d"). InnerVolumeSpecName "kube-api-access-h8rw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:03:19.528609 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.528592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0f58c1d-c595-415f-8809-adc720bece8d" (UID: "f0f58c1d-c595-415f-8809-adc720bece8d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:03:19.627636 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.627605    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rw2\" (UniqueName: \"kubernetes.io/projected/f0f58c1d-c595-415f-8809-adc720bece8d-kube-api-access-h8rw2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:03:19.627636 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.627635    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:03:19.627817 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.627645    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0f58c1d-c595-415f-8809-adc720bece8d-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:03:19.627817 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.627654    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0f58c1d-c595-415f-8809-adc720bece8d-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:03:19.627817 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.627664    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0f58c1d-c595-415f-8809-adc720bece8d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:03:19.801361 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.801284    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:03:19.807646 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:19.807620    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-mg8b6"]
Sep 09 21:03:20.607871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:20.607827    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:20.608300 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:20.608267    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:21.611133 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:21.611080    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" path="/var/lib/kubelet/pods/f0f58c1d-c595-415f-8809-adc720bece8d/volumes"
Sep 09 21:03:23.808933 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.808888    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:03:23.809445 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809383    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="storage-initializer"
Sep 09 21:03:23.809445 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809404    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="storage-initializer"
Sep 09 21:03:23.809445 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809419    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container"
Sep 09 21:03:23.809445 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809427    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container"
Sep 09 21:03:23.809659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kube-rbac-proxy"
Sep 09 21:03:23.809659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809465    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kube-rbac-proxy"
Sep 09 21:03:23.809659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809545    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kube-rbac-proxy"
Sep 09 21:03:23.809659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.809561    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f58c1d-c595-415f-8809-adc720bece8d" containerName="kserve-container"
Sep 09 21:03:23.904474 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.904431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:03:23.904640 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.904576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:23.907319 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.907296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:03:23.907462 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:23.907296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 09 21:03:24.069513 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.069440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.069513 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.069474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.069513 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.069510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.069875 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.069596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wwfs\" (UniqueName: \"kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.069875 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.069618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.170919 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.170877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.170919 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.170915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171307 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.170952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171307 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.170981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wwfs\" (UniqueName: \"kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171307 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.171008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171461 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.171393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171666 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.171635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.171784 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.171730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.173578 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.173562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.179554 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.179534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wwfs\" (UniqueName: \"kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.214399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.214371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:24.531630 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:24.531605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:03:24.533954 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:03:24.533919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9470a4d7_c0f8_4c73_bd91_6f6f4cda65dc.slice/crio-84622ebaca3118b5e5b1488bb0e99dd45979a518a9b9c9ec38e11a097cdcce80 WatchSource:0}: Error finding container 84622ebaca3118b5e5b1488bb0e99dd45979a518a9b9c9ec38e11a097cdcce80: Status 404 returned error can't find the container with id 84622ebaca3118b5e5b1488bb0e99dd45979a518a9b9c9ec38e11a097cdcce80
Sep 09 21:03:25.509767 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:25.509726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerStarted","Data":"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"}
Sep 09 21:03:25.510271 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:25.509836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerStarted","Data":"84622ebaca3118b5e5b1488bb0e99dd45979a518a9b9c9ec38e11a097cdcce80"}
Sep 09 21:03:29.528801 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:29.528764    2579 generic.go:356] "Generic (PLEG): container finished" podID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerID="5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047" exitCode=0
Sep 09 21:03:29.528801 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:29.528805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerDied","Data":"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"}
Sep 09 21:03:30.536701 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.536663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerStarted","Data":"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"}
Sep 09 21:03:30.537140 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.536737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerStarted","Data":"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"}
Sep 09 21:03:30.537140 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.536945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:30.559217 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.559160    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" podStartSLOduration=3.323677921 podStartE2EDuration="7.559142126s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.235464205s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:30.557874242 +0000 UTC m=+3143.394188706" watchObservedRunningTime="2026-09-09 21:03:30.559142126 +0000 UTC m=+3143.395456588"
Sep 09 21:03:30.606958 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.606916    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:30.607898 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.607861    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:30.657324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.657281    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:03:30.662469 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.662437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.665050 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.665016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 09 21:03:30.665187 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.665016    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 09 21:03:30.672026 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.671998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:03:30.724447 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.724397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.724447 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.724444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.724716 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.724461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp57g\" (UniqueName: \"kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.724716 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.724528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.724716 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.724587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825313 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825313 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825313 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp57g\" (UniqueName: \"kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825313 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825680 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.825680 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:03:30.825475    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 09 21:03:30.825680 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:03:30.825562    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls podName:166d1532-50a8-4b14-bf62-ca28b680b564 nodeName:}" failed. No retries permitted until 2026-09-09 21:03:31.32554008 +0000 UTC m=+3144.161854549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls") pod "isvc-pmml-predictor-8674795f59-jsg6w" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 09 21:03:30.825680 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.825664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.826173 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.826152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.826303 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.826227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:30.834580 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:30.834551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp57g\" (UniqueName: \"kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:31.330240 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:31.330196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:31.332838 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:31.332817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-jsg6w\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:31.541327 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:31.541293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:31.574639 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:31.574590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:31.856525 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:31.856493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:03:31.858975 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:03:31.858947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166d1532_50a8_4b14_bf62_ca28b680b564.slice/crio-9030ae4d70fe75fdadd26a975527251a811a45d0f4168de3c479715fde809b4a WatchSource:0}: Error finding container 9030ae4d70fe75fdadd26a975527251a811a45d0f4168de3c479715fde809b4a: Status 404 returned error can't find the container with id 9030ae4d70fe75fdadd26a975527251a811a45d0f4168de3c479715fde809b4a
Sep 09 21:03:32.547168 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:32.547125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerStarted","Data":"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"}
Sep 09 21:03:32.547621 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:32.547211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerStarted","Data":"9030ae4d70fe75fdadd26a975527251a811a45d0f4168de3c479715fde809b4a"}
Sep 09 21:03:36.566162 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:36.566119    2579 generic.go:356] "Generic (PLEG): container finished" podID="166d1532-50a8-4b14-bf62-ca28b680b564" containerID="2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4" exitCode=0
Sep 09 21:03:36.566538 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:36.566189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerDied","Data":"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"}
Sep 09 21:03:37.556233 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:37.556199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:03:40.607030 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:40.606985    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:40.607996 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:40.607965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:44.614207 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:44.614167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerStarted","Data":"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"}
Sep 09 21:03:44.614698 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:44.614245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerStarted","Data":"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"}
Sep 09 21:03:44.614698 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:44.614484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:44.637210 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:44.637151    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podStartSLOduration=3.458486262 podStartE2EDuration="14.637135303s" totalImagesPullingTime="7.34536075s" totalInitContainerRuntime="3.833288291s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:44.635293522 +0000 UTC m=+3157.471607988" watchObservedRunningTime="2026-09-09 21:03:44.637135303 +0000 UTC m=+3157.473449766"
Sep 09 21:03:45.619381 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:45.619343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:45.620772 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:45.620736    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:03:46.624065 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:46.624024    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:03:50.607624 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:50.607575    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:03:50.608176 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:50.608073    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:51.630436 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:51.630403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:03:51.631073 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:03:51.631042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:00.607676 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:00.607621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:00.608166 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:00.608141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:01.631485 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:01.631448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:07.561241 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:07.561212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:04:10.607142 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:10.607076    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:10.607999 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:10.607971    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:11.631021 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:11.630982    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:13.986398 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:13.986363    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:04:13.986853 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:13.986591    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kserve-container" containerID="cri-o://5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c" gracePeriod=30
Sep 09 21:04:13.986853 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:13.986652    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kube-rbac-proxy" containerID="cri-o://b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a" gracePeriod=30
Sep 09 21:04:14.747315 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:14.747276    2579 generic.go:356] "Generic (PLEG): container finished" podID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerID="b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a" exitCode=2
Sep 09 21:04:14.747496 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:14.747341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerDied","Data":"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"}
Sep 09 21:04:15.233688 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.233661    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:04:15.416784 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.416757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert\") pod \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") "
Sep 09 21:04:15.416988 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.416817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location\") pod \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") "
Sep 09 21:04:15.416988 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.416851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls\") pod \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") "
Sep 09 21:04:15.416988 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.416876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config\") pod \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") "
Sep 09 21:04:15.416988 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.416912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wwfs\" (UniqueName: \"kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs\") pod \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\" (UID: \"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc\") "
Sep 09 21:04:15.417244 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.417215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" (UID: "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:15.417305 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.417260    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" (UID: "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:15.417305 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.417283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" (UID: "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:15.419182 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.419153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" (UID: "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:15.419329 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.419306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs" (OuterVolumeSpecName: "kube-api-access-9wwfs") pod "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" (UID: "9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc"). InnerVolumeSpecName "kube-api-access-9wwfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:15.518430 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.518387    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:04:15.518430 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.518425    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:04:15.518430 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.518436    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:04:15.518680 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.518445    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:04:15.518680 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.518454    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wwfs\" (UniqueName: \"kubernetes.io/projected/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc-kube-api-access-9wwfs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:04:15.754219 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.754114    2579 generic.go:356] "Generic (PLEG): container finished" podID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerID="5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c" exitCode=0
Sep 09 21:04:15.754219 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.754198    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"
Sep 09 21:04:15.754456 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.754193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerDied","Data":"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"}
Sep 09 21:04:15.754456 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.754329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv" event={"ID":"9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc","Type":"ContainerDied","Data":"84622ebaca3118b5e5b1488bb0e99dd45979a518a9b9c9ec38e11a097cdcce80"}
Sep 09 21:04:15.754456 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.754369    2579 scope.go:117] "RemoveContainer" containerID="b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"
Sep 09 21:04:15.762609 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.762589    2579 scope.go:117] "RemoveContainer" containerID="5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"
Sep 09 21:04:15.770739 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.770720    2579 scope.go:117] "RemoveContainer" containerID="5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"
Sep 09 21:04:15.775899 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.775845    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:04:15.780165 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.780075    2579 scope.go:117] "RemoveContainer" containerID="b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"
Sep 09 21:04:15.780471 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:04:15.780445    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a\": container with ID starting with b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a not found: ID does not exist" containerID="b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"
Sep 09 21:04:15.780541 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.780485    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a"} err="failed to get container status \"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a\": rpc error: code = NotFound desc = could not find container \"b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a\": container with ID starting with b8e645306c11e366f0aaba7b6768cfd9a839310f0c9b82a43382c52201c4a72a not found: ID does not exist"
Sep 09 21:04:15.780541 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.780515    2579 scope.go:117] "RemoveContainer" containerID="5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"
Sep 09 21:04:15.780816 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:04:15.780796    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c\": container with ID starting with 5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c not found: ID does not exist" containerID="5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"
Sep 09 21:04:15.780891 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.780823    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c"} err="failed to get container status \"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c\": rpc error: code = NotFound desc = could not find container \"5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c\": container with ID starting with 5f646d0d269301a7b74c10dc19b2b0322f8ce7db4f02b8e17c50441b2c7d832c not found: ID does not exist"
Sep 09 21:04:15.780891 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.780846    2579 scope.go:117] "RemoveContainer" containerID="5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"
Sep 09 21:04:15.781188 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:04:15.781133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047\": container with ID starting with 5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047 not found: ID does not exist" containerID="5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"
Sep 09 21:04:15.781188 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.781169    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-zcbnv"]
Sep 09 21:04:15.781376 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:15.781168    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047"} err="failed to get container status \"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047\": rpc error: code = NotFound desc = could not find container \"5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047\": container with ID starting with 5a7b9fc1735fdf7fac96357da8d38c3d9c689becc42051afeed4e823a01d9047 not found: ID does not exist"
Sep 09 21:04:17.616202 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:17.616168    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" path="/var/lib/kubelet/pods/9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc/volumes"
Sep 09 21:04:20.607201 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:20.607156    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:20.607986 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:20.607960    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:21.631068 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:21.631026    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:30.606811 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:30.606761    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:30.607227 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:30.607042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:31.631435 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:31.631400    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:40.607450 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:40.607400    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:40.607960 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:40.607916    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:41.631771 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:41.631724    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:04:50.606715 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:50.606655    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:04:50.607180 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:50.607130    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:51.631283 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:04:51.631248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:05:00.607589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:00.607476    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:00.608153 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:00.607961    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:01.631466 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:01.631429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Sep 09 21:05:09.611363 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:09.611335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:05:10.607305 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:10.607261    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:10.607793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:10.607767    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:11.976911 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:11.976873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:05:11.977394 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:11.977167    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container" containerID="cri-o://2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af" gracePeriod=30
Sep 09 21:05:11.977394 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:11.977236    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kube-rbac-proxy" containerID="cri-o://4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96" gracePeriod=30
Sep 09 21:05:12.993324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:12.993290    2579 generic.go:356] "Generic (PLEG): container finished" podID="166d1532-50a8-4b14-bf62-ca28b680b564" containerID="4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96" exitCode=2
Sep 09 21:05:12.993711 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:12.993364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerDied","Data":"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"}
Sep 09 21:05:15.922956 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.922931    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:05:15.927015 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.926988    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert\") pod \"166d1532-50a8-4b14-bf62-ca28b680b564\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") "
Sep 09 21:05:15.927129 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location\") pod \"166d1532-50a8-4b14-bf62-ca28b680b564\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") "
Sep 09 21:05:15.927200 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config\") pod \"166d1532-50a8-4b14-bf62-ca28b680b564\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") "
Sep 09 21:05:15.927200 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp57g\" (UniqueName: \"kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g\") pod \"166d1532-50a8-4b14-bf62-ca28b680b564\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") "
Sep 09 21:05:15.927412 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "166d1532-50a8-4b14-bf62-ca28b680b564" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:15.927474 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "166d1532-50a8-4b14-bf62-ca28b680b564" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:15.927474 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927448    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:05:15.927589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.927495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "166d1532-50a8-4b14-bf62-ca28b680b564" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:15.929483 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:15.929457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g" (OuterVolumeSpecName: "kube-api-access-gp57g") pod "166d1532-50a8-4b14-bf62-ca28b680b564" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564"). InnerVolumeSpecName "kube-api-access-gp57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:16.008350 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.008311    2579 generic.go:356] "Generic (PLEG): container finished" podID="166d1532-50a8-4b14-bf62-ca28b680b564" containerID="2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af" exitCode=0
Sep 09 21:05:16.008536 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.008365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"
Sep 09 21:05:16.008536 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.008386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerDied","Data":"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"}
Sep 09 21:05:16.008536 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.008448    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w" event={"ID":"166d1532-50a8-4b14-bf62-ca28b680b564","Type":"ContainerDied","Data":"9030ae4d70fe75fdadd26a975527251a811a45d0f4168de3c479715fde809b4a"}
Sep 09 21:05:16.008536 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.008478    2579 scope.go:117] "RemoveContainer" containerID="4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"
Sep 09 21:05:16.016879 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.016862    2579 scope.go:117] "RemoveContainer" containerID="2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"
Sep 09 21:05:16.024662 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.024641    2579 scope.go:117] "RemoveContainer" containerID="2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"
Sep 09 21:05:16.027937 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.027913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") pod \"166d1532-50a8-4b14-bf62-ca28b680b564\" (UID: \"166d1532-50a8-4b14-bf62-ca28b680b564\") "
Sep 09 21:05:16.028122 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.028107    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/166d1532-50a8-4b14-bf62-ca28b680b564-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:05:16.028206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.028123    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp57g\" (UniqueName: \"kubernetes.io/projected/166d1532-50a8-4b14-bf62-ca28b680b564-kube-api-access-gp57g\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:05:16.028206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.028132    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/166d1532-50a8-4b14-bf62-ca28b680b564-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:05:16.029984 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.029950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "166d1532-50a8-4b14-bf62-ca28b680b564" (UID: "166d1532-50a8-4b14-bf62-ca28b680b564"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:16.032850 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.032828    2579 scope.go:117] "RemoveContainer" containerID="4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"
Sep 09 21:05:16.033185 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:16.033165    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96\": container with ID starting with 4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96 not found: ID does not exist" containerID="4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"
Sep 09 21:05:16.033269 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.033193    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96"} err="failed to get container status \"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96\": rpc error: code = NotFound desc = could not find container \"4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96\": container with ID starting with 4779289380ae36a7d5f80c4cecb0b6227829ca602f8515fad8ef1add42440a96 not found: ID does not exist"
Sep 09 21:05:16.033269 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.033212    2579 scope.go:117] "RemoveContainer" containerID="2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"
Sep 09 21:05:16.033483 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:16.033463    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af\": container with ID starting with 2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af not found: ID does not exist" containerID="2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"
Sep 09 21:05:16.033544 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.033492    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af"} err="failed to get container status \"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af\": rpc error: code = NotFound desc = could not find container \"2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af\": container with ID starting with 2528d7ee3fba477b83d45ad28e3dc8bc13aca5143b78c86a1384e321a22b54af not found: ID does not exist"
Sep 09 21:05:16.033544 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.033511    2579 scope.go:117] "RemoveContainer" containerID="2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"
Sep 09 21:05:16.033769 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:16.033753    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4\": container with ID starting with 2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4 not found: ID does not exist" containerID="2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"
Sep 09 21:05:16.033826 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.033772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4"} err="failed to get container status \"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4\": rpc error: code = NotFound desc = could not find container \"2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4\": container with ID starting with 2b2aa77bfe0aee39ae4f618551cb67a6843261afe2462dbe618c0ccc283ddff4 not found: ID does not exist"
Sep 09 21:05:16.128947 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.128909    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/166d1532-50a8-4b14-bf62-ca28b680b564-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:05:16.335971 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.335928    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:05:16.342615 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:16.342582    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-jsg6w"]
Sep 09 21:05:17.185878 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.185844    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186191    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="storage-initializer"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186203    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="storage-initializer"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186211    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186233    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186239    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186249    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="storage-initializer"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186255    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="storage-initializer"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186261    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kserve-container"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kserve-container"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186272    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186277    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kserve-container"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186329    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9470a4d7-c0f8-4c73-bd91-6f6f4cda65dc" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186335    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kube-rbac-proxy"
Sep 09 21:05:17.186576 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.186342    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" containerName="kserve-container"
Sep 09 21:05:17.190811 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.190793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.194283 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.194255    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:05:17.194392 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.194304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 09 21:05:17.215610 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.215579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:05:17.236334 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.236301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.236531 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.236345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.236531 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.236458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.236531 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.236495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjw2\" (UniqueName: \"kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.236677 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.236553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337130 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337130 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjw2\" (UniqueName: \"kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337431 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:17.337388    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 09 21:05:17.337624 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:17.337464    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls podName:a582fcbb-b287-4f2c-8b41-b9468cd3f9a5 nodeName:}" failed. No retries permitted until 2026-09-09 21:05:17.837442375 +0000 UTC m=+3250.673756831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-v5llc" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 09 21:05:17.337671 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337877 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.337916 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.337882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.361922 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.361895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjw2\" (UniqueName: \"kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.610606 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.610526    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166d1532-50a8-4b14-bf62-ca28b680b564" path="/var/lib/kubelet/pods/166d1532-50a8-4b14-bf62-ca28b680b564/volumes"
Sep 09 21:05:17.841015 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.840976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:17.843721 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:17.843695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-v5llc\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:18.101657 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:18.101617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:18.394666 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:18.394637    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:05:18.396963 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:05:18.396939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda582fcbb_b287_4f2c_8b41_b9468cd3f9a5.slice/crio-8185f3649c78082a83971dbbf49d0edf034f966292a87ef63139b37408896bbb WatchSource:0}: Error finding container 8185f3649c78082a83971dbbf49d0edf034f966292a87ef63139b37408896bbb: Status 404 returned error can't find the container with id 8185f3649c78082a83971dbbf49d0edf034f966292a87ef63139b37408896bbb
Sep 09 21:05:19.022999 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:19.022959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerStarted","Data":"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"}
Sep 09 21:05:19.023229 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:19.023032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerStarted","Data":"8185f3649c78082a83971dbbf49d0edf034f966292a87ef63139b37408896bbb"}
Sep 09 21:05:20.607275 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:20.607232    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:20.607728 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:20.607705    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:23.041694 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:23.041654    2579 generic.go:356] "Generic (PLEG): container finished" podID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerID="4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8" exitCode=0
Sep 09 21:05:23.042131 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:23.041734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerDied","Data":"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"}
Sep 09 21:05:24.049122 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.049065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerStarted","Data":"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"}
Sep 09 21:05:24.049512 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.049183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerStarted","Data":"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"}
Sep 09 21:05:24.049512 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.049453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:24.049605 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.049530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:24.050656 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.050629    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:05:24.082612 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:24.082550    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podStartSLOduration=3.412172796 podStartE2EDuration="7.082533474s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670360678s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:24.08184882 +0000 UTC m=+3256.918163282" watchObservedRunningTime="2026-09-09 21:05:24.082533474 +0000 UTC m=+3256.918847936"
Sep 09 21:05:25.053622 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:25.053579    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:05:30.058383 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:30.058352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:05:30.058906 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:30.058874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:05:30.606723 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:30.606666    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:30.607132 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:30.607106    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:40.059310 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:40.059266    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:05:40.607110 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:40.607049    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:40.607548 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:40.607518    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:44.526689 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.526643    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:05:44.531846 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.531822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.537003 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.536974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 09 21:05:44.537453 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.537435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 09 21:05:44.546052 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.546026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:05:44.627522 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.627480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.627522 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.627526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.627773 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.627624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppvr\" (UniqueName: \"kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.627773 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.627680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.627773 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.627700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.728998 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.728952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.728998 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729293 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729293 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729293 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppvr\" (UniqueName: \"kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729293 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:44.729215    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 09 21:05:44.729495 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:05:44.729300    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls podName:090fe57a-3ecb-470d-8051-64d4b9d49c0a nodeName:}" failed. No retries permitted until 2026-09-09 21:05:45.229278141 +0000 UTC m=+3278.065592584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-nrvlc" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 09 21:05:44.729570 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729707 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.729756 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.729738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:44.744279 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:44.744252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppvr\" (UniqueName: \"kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:45.234015 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.233979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:45.236669 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.236636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nrvlc\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:45.441528 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.441483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:45.606646 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.606593    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:45.607111 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.606839    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:45.740708 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:45.740683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:05:45.742325 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:05:45.742298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090fe57a_3ecb_470d_8051_64d4b9d49c0a.slice/crio-44450634efda415e45ad5d3d99fca428a1a53dce6cc4713525e910b53ffbfd98 WatchSource:0}: Error finding container 44450634efda415e45ad5d3d99fca428a1a53dce6cc4713525e910b53ffbfd98: Status 404 returned error can't find the container with id 44450634efda415e45ad5d3d99fca428a1a53dce6cc4713525e910b53ffbfd98
Sep 09 21:05:46.145869 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:46.145829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerStarted","Data":"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"}
Sep 09 21:05:46.146055 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:46.145911    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerStarted","Data":"44450634efda415e45ad5d3d99fca428a1a53dce6cc4713525e910b53ffbfd98"}
Sep 09 21:05:50.059776 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:50.059731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:05:52.171742 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:52.171706    2579 generic.go:356] "Generic (PLEG): container finished" podID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerID="06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b" exitCode=0
Sep 09 21:05:52.172295 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:52.171784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerDied","Data":"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"}
Sep 09 21:05:53.178926 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:53.178887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerStarted","Data":"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"}
Sep 09 21:05:53.179375 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:53.178954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerStarted","Data":"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"}
Sep 09 21:05:53.179375 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:53.179340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:53.205308 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:53.205258    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podStartSLOduration=3.353344437 podStartE2EDuration="9.205241861s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.851897424s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:53.204576575 +0000 UTC m=+3286.040891062" watchObservedRunningTime="2026-09-09 21:05:53.205241861 +0000 UTC m=+3286.041556324"
Sep 09 21:05:54.184147 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:54.184116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:05:54.185703 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:54.185669    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:05:55.188211 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:55.188173    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:05:55.607203 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:55.607077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:05:55.607562 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:05:55.607525    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:00.058852 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:00.058806    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:06:00.192793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:00.192742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:06:00.193427 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:00.193391    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:06:05.607161 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:05.607109    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:05.607705 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:05.607680    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:10.059492 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:10.059452    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:06:10.194001 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:10.193956    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:06:15.607249 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:15.607206    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:15.607758 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:15.607689    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:20.059398 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:20.059357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:06:20.193986 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:20.193942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:06:25.606514 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:25.606465    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:25.607292 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:25.606920    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:30.059208 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:30.059123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:06:30.194203 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:30.194155    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Sep 09 21:06:35.607007 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:35.606953    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:35.607524 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:35.607498    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:38.606800 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:38.606754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:06:40.194868 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:40.194837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:06:45.607333 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:45.607289    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:45.607878 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:45.607847    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:45.693165 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:45.693126    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:06:45.693438 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:45.693413    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container" containerID="cri-o://ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a" gracePeriod=30
Sep 09 21:06:45.693548 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:45.693499    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kube-rbac-proxy" containerID="cri-o://5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64" gracePeriod=30
Sep 09 21:06:46.412280 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:46.412248    2579 generic.go:356] "Generic (PLEG): container finished" podID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerID="5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64" exitCode=2
Sep 09 21:06:46.412473 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:46.412325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerDied","Data":"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"}
Sep 09 21:06:48.607487 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.607463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:06:48.734337 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.734305    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:06:48.781158 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config\") pod \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") "
Sep 09 21:06:48.781324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781174    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location\") pod \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") "
Sep 09 21:06:48.781324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert\") pod \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") "
Sep 09 21:06:48.781324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") pod \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") "
Sep 09 21:06:48.781324 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781302    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ppvr\" (UniqueName: \"kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr\") pod \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\" (UID: \"090fe57a-3ecb-470d-8051-64d4b9d49c0a\") "
Sep 09 21:06:48.781559 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "090fe57a-3ecb-470d-8051-64d4b9d49c0a" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:48.781661 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.781635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "090fe57a-3ecb-470d-8051-64d4b9d49c0a" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:48.783569 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.783527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr" (OuterVolumeSpecName: "kube-api-access-2ppvr") pod "090fe57a-3ecb-470d-8051-64d4b9d49c0a" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a"). InnerVolumeSpecName "kube-api-access-2ppvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:48.783681 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.783571    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "090fe57a-3ecb-470d-8051-64d4b9d49c0a" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:48.787409 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.787385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "090fe57a-3ecb-470d-8051-64d4b9d49c0a" (UID: "090fe57a-3ecb-470d-8051-64d4b9d49c0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:48.882946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.882850    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:06:48.882946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.882889    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/090fe57a-3ecb-470d-8051-64d4b9d49c0a-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:06:48.882946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.882900    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ppvr\" (UniqueName: \"kubernetes.io/projected/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-api-access-2ppvr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:06:48.882946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.882909    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:06:48.882946 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:48.882917    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/090fe57a-3ecb-470d-8051-64d4b9d49c0a-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:06:49.426230 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.426187    2579 generic.go:356] "Generic (PLEG): container finished" podID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerID="ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a" exitCode=0
Sep 09 21:06:49.426399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.426253    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"
Sep 09 21:06:49.426399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.426276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerDied","Data":"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"}
Sep 09 21:06:49.426399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.426359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc" event={"ID":"090fe57a-3ecb-470d-8051-64d4b9d49c0a","Type":"ContainerDied","Data":"44450634efda415e45ad5d3d99fca428a1a53dce6cc4713525e910b53ffbfd98"}
Sep 09 21:06:49.426399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.426387    2579 scope.go:117] "RemoveContainer" containerID="5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"
Sep 09 21:06:49.435653 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.435636    2579 scope.go:117] "RemoveContainer" containerID="ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"
Sep 09 21:06:49.443809 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.443786    2579 scope.go:117] "RemoveContainer" containerID="06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"
Sep 09 21:06:49.448802 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.448781    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:06:49.452241 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452221    2579 scope.go:117] "RemoveContainer" containerID="5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"
Sep 09 21:06:49.452321 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452305    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nrvlc"]
Sep 09 21:06:49.452549 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:06:49.452531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64\": container with ID starting with 5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64 not found: ID does not exist" containerID="5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"
Sep 09 21:06:49.452596 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452558    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64"} err="failed to get container status \"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64\": rpc error: code = NotFound desc = could not find container \"5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64\": container with ID starting with 5dce7b4e6f3143bbef65c1fb9636dd1b004c0e62a9b5906f302c82a6de590d64 not found: ID does not exist"
Sep 09 21:06:49.452596 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452577    2579 scope.go:117] "RemoveContainer" containerID="ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"
Sep 09 21:06:49.452892 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:06:49.452869    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a\": container with ID starting with ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a not found: ID does not exist" containerID="ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"
Sep 09 21:06:49.452934 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452893    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a"} err="failed to get container status \"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a\": rpc error: code = NotFound desc = could not find container \"ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a\": container with ID starting with ac27821eab3ad3dc6ba00b90368556e304f134b18397386ef0fd055bf50f9e8a not found: ID does not exist"
Sep 09 21:06:49.452934 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.452911    2579 scope.go:117] "RemoveContainer" containerID="06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"
Sep 09 21:06:49.453211 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:06:49.453188    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b\": container with ID starting with 06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b not found: ID does not exist" containerID="06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"
Sep 09 21:06:49.453279 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.453221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b"} err="failed to get container status \"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b\": rpc error: code = NotFound desc = could not find container \"06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b\": container with ID starting with 06f81bd9b47a74cb5f2558072418c4876b37519b3ed6bb55ea1e3e0c7bf56f6b not found: ID does not exist"
Sep 09 21:06:49.610502 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:49.610469    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" path="/var/lib/kubelet/pods/090fe57a-3ecb-470d-8051-64d4b9d49c0a/volumes"
Sep 09 21:06:55.606589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:55.606543    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:06:55.607027 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:55.606947    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:58.218122 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.218063    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:06:58.218599 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.218361    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" containerID="cri-o://b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd" gracePeriod=30
Sep 09 21:06:58.218599 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.218453    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kube-rbac-proxy" containerID="cri-o://e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c" gracePeriod=30
Sep 09 21:06:58.465567 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.465533    2579 generic.go:356] "Generic (PLEG): container finished" podID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerID="e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c" exitCode=2
Sep 09 21:06:58.465737 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.465572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerDied","Data":"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"}
Sep 09 21:06:58.607406 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:06:58.607311    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Sep 09 21:07:00.055921 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:00.055858    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Sep 09 21:07:01.956332 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:01.956303    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:07:02.100311 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config\") pod \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") "
Sep 09 21:07:02.100311 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert\") pod \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") "
Sep 09 21:07:02.100311 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location\") pod \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") "
Sep 09 21:07:02.100542 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") pod \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") "
Sep 09 21:07:02.100542 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gjw2\" (UniqueName: \"kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2\") pod \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\" (UID: \"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5\") "
Sep 09 21:07:02.100630 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100606    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:02.100679 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:02.100679 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:02.100797 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100778    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:07:02.100839 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100805    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:07:02.100839 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.100823    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:07:02.102734 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.102701    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:02.102734 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.102700    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2" (OuterVolumeSpecName: "kube-api-access-8gjw2") pod "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" (UID: "a582fcbb-b287-4f2c-8b41-b9468cd3f9a5"). InnerVolumeSpecName "kube-api-access-8gjw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:02.201450 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.201412    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:07:02.201450 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.201447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjw2\" (UniqueName: \"kubernetes.io/projected/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5-kube-api-access-8gjw2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:07:02.484470 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.484431    2579 generic.go:356] "Generic (PLEG): container finished" podID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerID="b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd" exitCode=0
Sep 09 21:07:02.484659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.484507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"
Sep 09 21:07:02.484659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.484511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerDied","Data":"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"}
Sep 09 21:07:02.484659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.484569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc" event={"ID":"a582fcbb-b287-4f2c-8b41-b9468cd3f9a5","Type":"ContainerDied","Data":"8185f3649c78082a83971dbbf49d0edf034f966292a87ef63139b37408896bbb"}
Sep 09 21:07:02.484659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.484596    2579 scope.go:117] "RemoveContainer" containerID="e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"
Sep 09 21:07:02.493625 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.493604    2579 scope.go:117] "RemoveContainer" containerID="b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"
Sep 09 21:07:02.501609 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.501589    2579 scope.go:117] "RemoveContainer" containerID="4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"
Sep 09 21:07:02.509459 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.509427    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:07:02.509965 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.509952    2579 scope.go:117] "RemoveContainer" containerID="e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"
Sep 09 21:07:02.510314 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:07:02.510291    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c\": container with ID starting with e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c not found: ID does not exist" containerID="e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"
Sep 09 21:07:02.510385 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.510321    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c"} err="failed to get container status \"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c\": rpc error: code = NotFound desc = could not find container \"e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c\": container with ID starting with e41950e306e0863585ff193c1b5c0f0a8bd575e46275d3d5030ee36c9c0f5f1c not found: ID does not exist"
Sep 09 21:07:02.510385 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.510340    2579 scope.go:117] "RemoveContainer" containerID="b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"
Sep 09 21:07:02.510608 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:07:02.510591    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd\": container with ID starting with b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd not found: ID does not exist" containerID="b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"
Sep 09 21:07:02.510648 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.510611    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd"} err="failed to get container status \"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd\": rpc error: code = NotFound desc = could not find container \"b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd\": container with ID starting with b6385f568b33aaa8410018c78541b0b0f2f120db4123b5456153ac277032b4cd not found: ID does not exist"
Sep 09 21:07:02.510648 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.510627    2579 scope.go:117] "RemoveContainer" containerID="4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"
Sep 09 21:07:02.510891 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:07:02.510873    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8\": container with ID starting with 4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8 not found: ID does not exist" containerID="4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"
Sep 09 21:07:02.510948 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.510894    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8"} err="failed to get container status \"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8\": rpc error: code = NotFound desc = could not find container \"4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8\": container with ID starting with 4c9b332368278ac7a8dfcc2b3fbf7f0ce49e5adc639016de5725fc1614e557f8 not found: ID does not exist"
Sep 09 21:07:02.517166 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:02.517142    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-v5llc"]
Sep 09 21:07:03.384902 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.384868    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385181    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="storage-initializer"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385194    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="storage-initializer"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385226    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="storage-initializer"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385231    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="storage-initializer"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385243    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385248    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385253    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385260    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container"
Sep 09 21:07:03.385291 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container"
Sep 09 21:07:03.385659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385307    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kserve-container"
Sep 09 21:07:03.385659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385316    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385322    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" containerName="kube-rbac-proxy"
Sep 09 21:07:03.385659 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.385327    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="090fe57a-3ecb-470d-8051-64d4b9d49c0a" containerName="kserve-container"
Sep 09 21:07:03.389834 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.389816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.392387 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.392365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 09 21:07:03.392521 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.392392    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 21:07:03.404146 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.404123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:07:03.513858 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.513821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsk7\" (UniqueName: \"kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.514012 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.513864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.514012 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.513888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.514012 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.513956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.514170 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.514026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.610515 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.610482    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a582fcbb-b287-4f2c-8b41-b9468cd3f9a5" path="/var/lib/kubelet/pods/a582fcbb-b287-4f2c-8b41-b9468cd3f9a5/volumes"
Sep 09 21:07:03.614761 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.614741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.614845 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.614790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.614845 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.614821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsk7\" (UniqueName: \"kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.614845 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.614842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.615019 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.614860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.615359 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.615334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.615629 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.615610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.615952 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.615932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.617529 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.617505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.623817 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.623795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsk7\" (UniqueName: \"kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.700026 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.699986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:03.974330 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:03.974246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:07:03.977760 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:07:03.977731    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7edd6ba_ce7f_4e25_b4be_f9cc931d9f07.slice/crio-ef166c673c46a6dcde463a275544ceaf00ab9b328f160c3794d715155ef43b0a WatchSource:0}: Error finding container ef166c673c46a6dcde463a275544ceaf00ab9b328f160c3794d715155ef43b0a: Status 404 returned error can't find the container with id ef166c673c46a6dcde463a275544ceaf00ab9b328f160c3794d715155ef43b0a
Sep 09 21:07:04.495381 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:04.495343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerStarted","Data":"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"}
Sep 09 21:07:04.495793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:04.495411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerStarted","Data":"ef166c673c46a6dcde463a275544ceaf00ab9b328f160c3794d715155ef43b0a"}
Sep 09 21:07:05.606655 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:05.606584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:05.607228 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:05.607193    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:07.611790 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:07.611746    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:07.612249 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:07.612008    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:08.513429 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:08.513396    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerID="1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f" exitCode=0
Sep 09 21:07:08.513646 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:08.513482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerDied","Data":"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"}
Sep 09 21:07:09.520962 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.520925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerStarted","Data":"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"}
Sep 09 21:07:09.523489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.521016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerStarted","Data":"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"}
Sep 09 21:07:09.523489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.521320    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:09.523489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.521381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:09.523489 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.522921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:09.541241 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:09.541187    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podStartSLOduration=2.7552821400000003 podStartE2EDuration="6.541171889s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785889749s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:09.539307237 +0000 UTC m=+3362.375621714" watchObservedRunningTime="2026-09-09 21:07:09.541171889 +0000 UTC m=+3362.377486353"
Sep 09 21:07:10.525406 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:10.525367    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:15.529990 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:15.529961    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:07:15.530523 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:15.530495    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:17.612099 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:17.612035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:17.612566 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:17.612542    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:25.531306 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:25.531262    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:27.611801 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:27.611758    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:27.612397 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:27.612370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:35.531458 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:35.531405    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:37.611748 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:37.611695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:37.612269 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:37.612196    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:45.530683 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:45.530641    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:47.612154 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:47.612110    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:47.612645 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:47.612622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:55.530654 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:55.530607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:07:57.612458 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:57.612418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:07:57.612922 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:07:57.612899    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:05.531049 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:05.530954    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:08:07.611717 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:07.611674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:07.612224 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:07.612199    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:12.606563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:12.606518    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:12.607032 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:12.607003    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:15.530877 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:15.530836    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:08:22.606564 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:22.606508    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:22.607028 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:22.606953    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:25.531357 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:25.531309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:08:32.606996 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:32.606947    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:32.607554 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:32.607521    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:35.531248 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:35.531216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:08:36.135117 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.135066    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:36.138556 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.138539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.141184 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.141161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f4eae3-predictor-serving-cert\""
Sep 09 21:08:36.141300 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.141185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f4eae3\""
Sep 09 21:08:36.141300 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.141243    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f4eae3-kube-rbac-proxy-sar-config\""
Sep 09 21:08:36.142099 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.142072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f4eae3-dockercfg-s582g\""
Sep 09 21:08:36.149201 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.149176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:36.204333 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.204297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.204333 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.204347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.204566 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.204376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.204566 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.204444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.204566 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.204507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9lf\" (UniqueName: \"kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305149 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9lf\" (UniqueName: \"kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305689 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305822 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.305883 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.305867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.307871 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.307852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.314220 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.314198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9lf\" (UniqueName: \"kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf\") pod \"isvc-secondary-f4eae3-predictor-647fdf646-m9dsw\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") " pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.449546 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.449509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:36.724910 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.724883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:36.727453 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:08:36.727424    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a63cda_6ecc_4a1f_9675_a1ae7f0572bd.slice/crio-30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9 WatchSource:0}: Error finding container 30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9: Status 404 returned error can't find the container with id 30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9
Sep 09 21:08:36.881622 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.881585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerStarted","Data":"a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"}
Sep 09 21:08:36.881818 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:36.881644    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerStarted","Data":"30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9"}
Sep 09 21:08:42.606471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:42.606413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:42.606985 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:42.606847    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:42.907099 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:42.907065    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/0.log"
Sep 09 21:08:42.907280 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:42.907132    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerID="a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec" exitCode=1
Sep 09 21:08:42.907280 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:42.907209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerDied","Data":"a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"}
Sep 09 21:08:43.916812 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:43.916783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/0.log"
Sep 09 21:08:43.917257 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:43.916866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerStarted","Data":"2cb00103da372e22270a1cccfb9638ce36ecf7e65d197a8100b5238ae05aa254"}
Sep 09 21:08:44.526026 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:44.525991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:08:44.526297 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:44.526270    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" containerID="cri-o://467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a" gracePeriod=30
Sep 09 21:08:44.526348 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:44.526294    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kube-rbac-proxy" containerID="cri-o://2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea" gracePeriod=30
Sep 09 21:08:44.924014 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:44.923984    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerID="2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea" exitCode=2
Sep 09 21:08:44.924407 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:44.924019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerDied","Data":"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"}
Sep 09 21:08:45.526339 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:45.526285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Sep 09 21:08:45.530565 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:45.530537    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Sep 09 21:08:46.933005 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.932978    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/1.log"
Sep 09 21:08:46.933431 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.933402    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/0.log"
Sep 09 21:08:46.933478 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.933439    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerID="2cb00103da372e22270a1cccfb9638ce36ecf7e65d197a8100b5238ae05aa254" exitCode=1
Sep 09 21:08:46.933478 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.933469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerDied","Data":"2cb00103da372e22270a1cccfb9638ce36ecf7e65d197a8100b5238ae05aa254"}
Sep 09 21:08:46.933547 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.933537    2579 scope.go:117] "RemoveContainer" containerID="a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"
Sep 09 21:08:46.933955 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.933935    2579 scope.go:117] "RemoveContainer" containerID="a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"
Sep 09 21:08:46.944763 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:08:46.944738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_kserve-ci-e2e-test(b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd)\"" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"
Sep 09 21:08:46.944939 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:08:46.944908    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_kserve-ci-e2e-test_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd_0 in pod sandbox 30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9 from index: no such id: 'a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec'" containerID="a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"
Sep 09 21:08:46.945007 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:46.944947    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_kserve-ci-e2e-test_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd_0 in pod sandbox 30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9 from index: no such id: 'a60fa167a1d9a1cbf9cd43daebcb75731f2529616f4fcc355490fc8917acc5ec'"
Sep 09 21:08:47.938905 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:47.938879    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/1.log"
Sep 09 21:08:48.366271 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.366245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:08:48.517785 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.517701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location\") pod \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") "
Sep 09 21:08:48.517785 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.517746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert\") pod \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") "
Sep 09 21:08:48.517785 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.517771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config\") pod \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") "
Sep 09 21:08:48.518114 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.517814    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls\") pod \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") "
Sep 09 21:08:48.518114 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.517850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djsk7\" (UniqueName: \"kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7\") pod \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\" (UID: \"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07\") "
Sep 09 21:08:48.518114 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.518062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" (UID: "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:48.518275 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.518213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" (UID: "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:48.518275 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.518221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" (UID: "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:48.520144 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.520119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" (UID: "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:48.520270 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.520246    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7" (OuterVolumeSpecName: "kube-api-access-djsk7") pod "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" (UID: "c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07"). InnerVolumeSpecName "kube-api-access-djsk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:48.618471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.618434    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:48.618471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.618463    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:48.618471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.618474    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:48.618471 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.618483    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:48.618747 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.618493    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsk7\" (UniqueName: \"kubernetes.io/projected/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07-kube-api-access-djsk7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:48.945638 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.945599    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerID="467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a" exitCode=0
Sep 09 21:08:48.946148 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.945664    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"
Sep 09 21:08:48.946148 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.945681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerDied","Data":"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"}
Sep 09 21:08:48.946148 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.945738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf" event={"ID":"c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07","Type":"ContainerDied","Data":"ef166c673c46a6dcde463a275544ceaf00ab9b328f160c3794d715155ef43b0a"}
Sep 09 21:08:48.946148 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.945763    2579 scope.go:117] "RemoveContainer" containerID="2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"
Sep 09 21:08:48.954286 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.954268    2579 scope.go:117] "RemoveContainer" containerID="467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"
Sep 09 21:08:48.962141 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.962118    2579 scope.go:117] "RemoveContainer" containerID="1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"
Sep 09 21:08:48.969214 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.969192    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:08:48.969924 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.969904    2579 scope.go:117] "RemoveContainer" containerID="2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"
Sep 09 21:08:48.970285 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:08:48.970256    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea\": container with ID starting with 2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea not found: ID does not exist" containerID="2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"
Sep 09 21:08:48.970381 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.970297    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea"} err="failed to get container status \"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea\": rpc error: code = NotFound desc = could not find container \"2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea\": container with ID starting with 2f7dfb3b333ebb6b452417b8b1bb3e2324a5e66a81bba5bc7a3d384a85fd05ea not found: ID does not exist"
Sep 09 21:08:48.970381 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.970317    2579 scope.go:117] "RemoveContainer" containerID="467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"
Sep 09 21:08:48.970604 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:08:48.970574    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a\": container with ID starting with 467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a not found: ID does not exist" containerID="467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"
Sep 09 21:08:48.970645 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.970610    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a"} err="failed to get container status \"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a\": rpc error: code = NotFound desc = could not find container \"467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a\": container with ID starting with 467a9afebab2bd5c1157adaee51ad3b54da24099553aed34b3e653cef2a5488a not found: ID does not exist"
Sep 09 21:08:48.970645 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.970627    2579 scope.go:117] "RemoveContainer" containerID="1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"
Sep 09 21:08:48.970892 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:08:48.970876    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f\": container with ID starting with 1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f not found: ID does not exist" containerID="1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"
Sep 09 21:08:48.970935 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.970899    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f"} err="failed to get container status \"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f\": rpc error: code = NotFound desc = could not find container \"1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f\": container with ID starting with 1608d1cc89326231d1bf96b62d88a80284e0dd03ced9a75871c6bf0c8ec8750f not found: ID does not exist"
Sep 09 21:08:48.975623 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:48.975604    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-jg2lf"]
Sep 09 21:08:49.610821 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:49.610790    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" path="/var/lib/kubelet/pods/c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07/volumes"
Sep 09 21:08:52.606619 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:52.606569    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:08:52.607045 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:52.607013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:08:54.177643 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.177609    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:54.305547 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.305526    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/1.log"
Sep 09 21:08:54.307905 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.307883    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:54.470591 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls\") pod \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") "
Sep 09 21:08:54.470591 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config\") pod \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") "
Sep 09 21:08:54.470591 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location\") pod \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") "
Sep 09 21:08:54.470872 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert\") pod \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") "
Sep 09 21:08:54.470872 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td9lf\" (UniqueName: \"kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf\") pod \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\" (UID: \"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd\") "
Sep 09 21:08:54.470993 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" (UID: "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:54.470993 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.470954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" (UID: "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:54.471204 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.471071    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" (UID: "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:54.472857 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.472827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" (UID: "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:54.472971 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.472865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf" (OuterVolumeSpecName: "kube-api-access-td9lf") pod "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" (UID: "b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd"). InnerVolumeSpecName "kube-api-access-td9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:54.571830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.571779    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-cabundle-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:54.571830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.571825    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9lf\" (UniqueName: \"kubernetes.io/projected/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-api-access-td9lf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:54.571830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.571835    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-proxy-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:54.571830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.571845    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:54.571830 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.571852    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd-kserve-provision-location\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:08:54.973819 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.973790    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f4eae3-predictor-647fdf646-m9dsw_b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/storage-initializer/1.log"
Sep 09 21:08:54.974020 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.973885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw" event={"ID":"b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd","Type":"ContainerDied","Data":"30d4f8df883bca89ea05cc194f5a2ffa658370871d7cb47b187c34bcf1c055c9"}
Sep 09 21:08:54.974020 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.973900    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"
Sep 09 21:08:54.974020 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:54.973961    2579 scope.go:117] "RemoveContainer" containerID="2cb00103da372e22270a1cccfb9638ce36ecf7e65d197a8100b5238ae05aa254"
Sep 09 21:08:55.014910 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:55.014879    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:55.020181 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:55.020065    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f4eae3-predictor-647fdf646-m9dsw"]
Sep 09 21:08:55.612018 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:08:55.611982    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" path="/var/lib/kubelet/pods/b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd/volumes"
Sep 09 21:09:00.820770 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.820731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ptfbq/must-gather-gmzcj"]
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821109    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821123    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kube-rbac-proxy"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kube-rbac-proxy"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="storage-initializer"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821162    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="storage-initializer"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821167    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821206 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821185    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821234    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kserve-container"
Sep 09 21:09:00.821563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821244    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821249    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a63cda-6ecc-4a1f-9675-a1ae7f0572bd" containerName="storage-initializer"
Sep 09 21:09:00.821563 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.821254    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7edd6ba-ce7f-4e25-b4be-f9cc931d9f07" containerName="kube-rbac-proxy"
Sep 09 21:09:00.825585 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.825563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:00.828309 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.828286    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ptfbq\"/\"kube-root-ca.crt\""
Sep 09 21:09:00.828455 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.828314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ptfbq\"/\"openshift-service-ca.crt\""
Sep 09 21:09:00.834973 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.834944    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptfbq/must-gather-gmzcj"]
Sep 09 21:09:00.916607 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.916569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:00.916793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:00.916637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvb2\" (UniqueName: \"kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.017382 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.017333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.017564 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.017424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvb2\" (UniqueName: \"kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.017677 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.017658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.026472 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.026452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvb2\" (UniqueName: \"kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2\") pod \"must-gather-gmzcj\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") " pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.143598 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.143566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:01.414843 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.414813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptfbq/must-gather-gmzcj"]
Sep 09 21:09:01.416636 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:09:01.416610    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda508d9d9_232f_4dfe_90fc_283c72d1dd4c.slice/crio-c41939e64a2be521d17c817f5439b7d054c81a2e9fc8b971239b67e8161eeff4 WatchSource:0}: Error finding container c41939e64a2be521d17c817f5439b7d054c81a2e9fc8b971239b67e8161eeff4: Status 404 returned error can't find the container with id c41939e64a2be521d17c817f5439b7d054c81a2e9fc8b971239b67e8161eeff4
Sep 09 21:09:01.418398 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:01.418377    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:09:02.004125 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:02.004071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptfbq/must-gather-gmzcj" event={"ID":"a508d9d9-232f-4dfe-90fc-283c72d1dd4c","Type":"ContainerStarted","Data":"c41939e64a2be521d17c817f5439b7d054c81a2e9fc8b971239b67e8161eeff4"}
Sep 09 21:09:02.606501 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:02.606414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:02.607016 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:02.606969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:08.041037 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:08.040995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptfbq/must-gather-gmzcj" event={"ID":"a508d9d9-232f-4dfe-90fc-283c72d1dd4c","Type":"ContainerStarted","Data":"7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39"}
Sep 09 21:09:08.041450 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:08.041051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptfbq/must-gather-gmzcj" event={"ID":"a508d9d9-232f-4dfe-90fc-283c72d1dd4c","Type":"ContainerStarted","Data":"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"}
Sep 09 21:09:12.606448 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:12.606401    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:12.607156 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:12.607129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:16.864516 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:16.864480    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:16.888809 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:16.888779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:16.903150 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:16.903111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:16.911547 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:16.911525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:18.328075 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:18.328044    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:18.342665 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:18.342635    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:18.358619 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:18.358592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:18.366529 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:18.366508    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:19.745138 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:19.745107    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:19.757079 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:19.757051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:19.770986 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:19.770963    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:19.778281 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:19.778258    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:21.086722 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:21.086695    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:21.098723 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:21.098695    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:21.113398 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:21.113375    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:21.121336 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:21.121313    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:22.476129 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.476100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:22.489866 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.489840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:22.504697 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.504653    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:22.511926 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.511898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:22.607723 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.607672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:22.608281 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:22.608249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:23.858706 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:23.858672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:23.870493 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:23.870466    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:23.884367 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:23.884339    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:23.892955 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:23.892924    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:25.236251 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:25.236222    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:25.248895 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:25.248866    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:25.265398 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:25.265372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:25.276752 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:25.276728    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:26.632033 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:26.632007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:26.645787 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:26.645764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:26.660480 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:26.660450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:26.667851 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:26.667828    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:27.983771 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:27.983745    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:27.996205 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:27.996174    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:28.012261 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:28.012225    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:28.019796 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:28.019770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:29.433389 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:29.433305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:29.446726 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:29.446700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:29.462353 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:29.462324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:29.469478 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:29.469431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:31.084867 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:31.084836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:31.097447 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:31.097424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:31.112352 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:31.112324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:31.120507 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:31.120424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:32.485627 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.485587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:32.499311 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.499274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:32.520783 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.520757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:32.527889 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.527867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:32.607399 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.607347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:32.608036 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:32.608005    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:33.606814 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.606762    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:33.607314 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.607043    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:33.870459 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.870364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:33.882623 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.882594    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:33.896764 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.896739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:33.905709 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:33.905688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:35.320504 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:35.320475    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:35.355493 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:35.355463    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:35.381348 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:35.381322    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:35.406389 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:35.406360    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:36.830523 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:36.830493    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kserve-container/0.log"
Sep 09 21:09:36.844942 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:36.844914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/kube-rbac-proxy/0.log"
Sep 09 21:09:36.862016 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:36.861986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/agent/0.log"
Sep 09 21:09:36.871788 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:36.871765    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5_6213c661-f8ab-48cd-982c-75851ecf0583/storage-initializer/0.log"
Sep 09 21:09:38.182507 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:38.182469    2579 generic.go:356] "Generic (PLEG): container finished" podID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerID="e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd" exitCode=0
Sep 09 21:09:38.182951 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:38.182533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptfbq/must-gather-gmzcj" event={"ID":"a508d9d9-232f-4dfe-90fc-283c72d1dd4c","Type":"ContainerDied","Data":"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"}
Sep 09 21:09:38.182951 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:38.182839    2579 scope.go:117] "RemoveContainer" containerID="e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"
Sep 09 21:09:39.128046 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:39.128010    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ptfbq_must-gather-gmzcj_a508d9d9-232f-4dfe-90fc-283c72d1dd4c/gather/0.log"
Sep 09 21:09:43.148786 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:43.148746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wqftt_7b059ae5-251d-4a2b-a6e2-81571644fa39/global-pull-secret-syncer/0.log"
Sep 09 21:09:43.256690 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:43.256652    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q8jz5_3d7ad2e7-1636-49d4-be9a-cade8d13b83e/konnectivity-agent/0.log"
Sep 09 21:09:43.366423 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:43.366392    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-45.ec2.internal_1034d32653d0ee36541005678f807d7f/haproxy/0.log"
Sep 09 21:09:43.606981 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:43.606857    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:43.607369 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:43.607341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:44.742808 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:44.742768    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ptfbq/must-gather-gmzcj"]
Sep 09 21:09:44.743305 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:44.742962    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-ptfbq/must-gather-gmzcj" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="copy" containerID="cri-o://7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39" gracePeriod=2
Sep 09 21:09:44.747653 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:44.747349    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ptfbq/must-gather-gmzcj"]
Sep 09 21:09:44.979757 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:44.979735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ptfbq_must-gather-gmzcj_a508d9d9-232f-4dfe-90fc-283c72d1dd4c/copy/0.log"
Sep 09 21:09:44.982343 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:44.982325    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:44.986860 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:09:44.986828    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gmzcj\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ptfbq\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:45.101739 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.101654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output\") pod \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") "
Sep 09 21:09:45.101894 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.101758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvb2\" (UniqueName: \"kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2\") pod \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\" (UID: \"a508d9d9-232f-4dfe-90fc-283c72d1dd4c\") "
Sep 09 21:09:45.103926 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.103897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "a508d9d9-232f-4dfe-90fc-283c72d1dd4c" (UID: "a508d9d9-232f-4dfe-90fc-283c72d1dd4c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:45.104156 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.104128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2" (OuterVolumeSpecName: "kube-api-access-6bvb2") pod "a508d9d9-232f-4dfe-90fc-283c72d1dd4c" (UID: "a508d9d9-232f-4dfe-90fc-283c72d1dd4c"). InnerVolumeSpecName "kube-api-access-6bvb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:45.202704 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.202665    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvb2\" (UniqueName: \"kubernetes.io/projected/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-kube-api-access-6bvb2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:09:45.202704 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.202700    2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a508d9d9-232f-4dfe-90fc-283c72d1dd4c-must-gather-output\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Sep 09 21:09:45.217250 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.217225    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ptfbq_must-gather-gmzcj_a508d9d9-232f-4dfe-90fc-283c72d1dd4c/copy/0.log"
Sep 09 21:09:45.217615 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.217593    2579 generic.go:356] "Generic (PLEG): container finished" podID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerID="7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39" exitCode=143
Sep 09 21:09:45.217707 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.217639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:45.217707 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.217673    2579 scope.go:117] "RemoveContainer" containerID="7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39"
Sep 09 21:09:45.221290 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:09:45.221256    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gmzcj\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ptfbq\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:45.226111 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.226074    2579 scope.go:117] "RemoveContainer" containerID="e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"
Sep 09 21:09:45.228660 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:09:45.228636    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gmzcj\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ptfbq\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" pod="openshift-must-gather-ptfbq/must-gather-gmzcj"
Sep 09 21:09:45.239770 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.239747    2579 scope.go:117] "RemoveContainer" containerID="7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39"
Sep 09 21:09:45.240078 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:09:45.240061    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39\": container with ID starting with 7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39 not found: ID does not exist" containerID="7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39"
Sep 09 21:09:45.240143 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.240101    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39"} err="failed to get container status \"7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39\": rpc error: code = NotFound desc = could not find container \"7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39\": container with ID starting with 7aab2aa2ea0f6469fb762312d6f4cadced634a0808c0e751189dc5a55798eb39 not found: ID does not exist"
Sep 09 21:09:45.240143 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.240120    2579 scope.go:117] "RemoveContainer" containerID="e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"
Sep 09 21:09:45.240440 ip-10-0-129-45 kubenswrapper[2579]: E0909 21:09:45.240417    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd\": container with ID starting with e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd not found: ID does not exist" containerID="e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"
Sep 09 21:09:45.240543 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.240448    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd"} err="failed to get container status \"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd\": rpc error: code = NotFound desc = could not find container \"e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd\": container with ID starting with e1e9d3b9312b8a65eb25103ebc5d96416a18cd70a734c7814e3206affb7493dd not found: ID does not exist"
Sep 09 21:09:45.611874 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:45.611840    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" path="/var/lib/kubelet/pods/a508d9d9-232f-4dfe-90fc-283c72d1dd4c/volumes"
Sep 09 21:09:47.981644 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:47.981570    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xwmc2_ed1ae4a7-c766-4d53-8f17-70ec080d2c71/kube-state-metrics/0.log"
Sep 09 21:09:48.004365 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.004338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xwmc2_ed1ae4a7-c766-4d53-8f17-70ec080d2c71/kube-rbac-proxy-main/0.log"
Sep 09 21:09:48.029581 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.029550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-xwmc2_ed1ae4a7-c766-4d53-8f17-70ec080d2c71/kube-rbac-proxy-self/0.log"
Sep 09 21:09:48.096473 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.096448    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-vlc44_8ae15eb3-7466-40fb-9665-54a46938bc15/monitoring-plugin/0.log"
Sep 09 21:09:48.128673 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.128645    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4p6zm_9a5f94bb-01d4-4fdf-86e4-ec6eb126be78/node-exporter/0.log"
Sep 09 21:09:48.159925 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.159903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4p6zm_9a5f94bb-01d4-4fdf-86e4-ec6eb126be78/kube-rbac-proxy/0.log"
Sep 09 21:09:48.187113 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.187067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4p6zm_9a5f94bb-01d4-4fdf-86e4-ec6eb126be78/init-textfile/0.log"
Sep 09 21:09:48.468065 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.468038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hzblv_dae08fee-6d37-4125-a77f-19e233dc3849/kube-rbac-proxy-main/0.log"
Sep 09 21:09:48.494769 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.494736    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hzblv_dae08fee-6d37-4125-a77f-19e233dc3849/kube-rbac-proxy-self/0.log"
Sep 09 21:09:48.519173 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.519144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-hzblv_dae08fee-6d37-4125-a77f-19e233dc3849/openshift-state-metrics/0.log"
Sep 09 21:09:48.559042 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.558995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/prometheus/0.log"
Sep 09 21:09:48.583821 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.583795    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/config-reloader/0.log"
Sep 09 21:09:48.617638 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.617611    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/thanos-sidecar/0.log"
Sep 09 21:09:48.641313 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.641287    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/kube-rbac-proxy-web/0.log"
Sep 09 21:09:48.669497 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.669474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/kube-rbac-proxy/0.log"
Sep 09 21:09:48.694187 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.694162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/kube-rbac-proxy-thanos/0.log"
Sep 09 21:09:48.719763 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.719698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_785c8670-0a48-4255-9a8b-a80764a3eed9/init-config-reloader/0.log"
Sep 09 21:09:48.850734 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.850702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56fcf65b95-hvjv4_8a4bb0f4-5fa8-4554-aa77-af443b5f74a8/telemeter-client/0.log"
Sep 09 21:09:48.873368 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.873344    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56fcf65b95-hvjv4_8a4bb0f4-5fa8-4554-aa77-af443b5f74a8/reload/0.log"
Sep 09 21:09:48.895316 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:48.895291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56fcf65b95-hvjv4_8a4bb0f4-5fa8-4554-aa77-af443b5f74a8/kube-rbac-proxy/0.log"
Sep 09 21:09:51.386189 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"]
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386487    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="gather"
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="gather"
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="copy"
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="copy"
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386571    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="gather"
Sep 09 21:09:51.386600 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.386589    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a508d9d9-232f-4dfe-90fc-283c72d1dd4c" containerName="copy"
Sep 09 21:09:51.391956 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.391929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.394526 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.394498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9tpv\"/\"openshift-service-ca.crt\""
Sep 09 21:09:51.394682 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.394502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w9tpv\"/\"default-dockercfg-7bnrw\""
Sep 09 21:09:51.395487 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.395469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w9tpv\"/\"kube-root-ca.crt\""
Sep 09 21:09:51.399345 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.399323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"]
Sep 09 21:09:51.456811 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.456766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-lib-modules\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.456994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.456819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-podres\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.456994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.456849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgf8\" (UniqueName: \"kubernetes.io/projected/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-kube-api-access-hhgf8\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.456994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.456925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-sys\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.456994 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.456957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-proc\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557386 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-lib-modules\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-podres\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgf8\" (UniqueName: \"kubernetes.io/projected/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-kube-api-access-hhgf8\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-sys\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-proc\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-podres\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-sys\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557822 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-proc\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.557822 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.557545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-lib-modules\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.567032 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.567002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgf8\" (UniqueName: \"kubernetes.io/projected/a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41-kube-api-access-hhgf8\") pod \"perf-node-gather-daemonset-kwqzf\" (UID: \"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41\") " pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.702414 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.702372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:51.974269 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:51.974240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"]
Sep 09 21:09:51.976827 ip-10-0-129-45 kubenswrapper[2579]: W0909 21:09:51.976799    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda7e0dd87_e38d_4bb0_b91f_db64b8e2dc41.slice/crio-878ba5a9c4649dcd24c08d482ff99520d793430feaedc1ebd79050d2dbdd7192 WatchSource:0}: Error finding container 878ba5a9c4649dcd24c08d482ff99520d793430feaedc1ebd79050d2dbdd7192: Status 404 returned error can't find the container with id 878ba5a9c4649dcd24c08d482ff99520d793430feaedc1ebd79050d2dbdd7192
Sep 09 21:09:52.250979 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:52.250881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf" event={"ID":"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41","Type":"ContainerStarted","Data":"5ff73b200eab320fb255df89930711be80988f4b1a56036f61987783826aaa23"}
Sep 09 21:09:52.250979 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:52.250932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf" event={"ID":"a7e0dd87-e38d-4bb0-b91f-db64b8e2dc41","Type":"ContainerStarted","Data":"878ba5a9c4649dcd24c08d482ff99520d793430feaedc1ebd79050d2dbdd7192"}
Sep 09 21:09:52.251275 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:52.251014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:52.271604 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:52.271545    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf" podStartSLOduration=1.271529707 podStartE2EDuration="1.271529707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:52.270209577 +0000 UTC m=+3525.106524055" watchObservedRunningTime="2026-09-09 21:09:52.271529707 +0000 UTC m=+3525.107844170"
Sep 09 21:09:53.052202 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.052172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pn5rp_09e49098-9703-4ac2-97a4-66d87c1084da/dns/0.log"
Sep 09 21:09:53.077063 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.077034    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pn5rp_09e49098-9703-4ac2-97a4-66d87c1084da/kube-rbac-proxy/0.log"
Sep 09 21:09:53.284828 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.284802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xt4vg_67aea7a1-9c4a-4bb3-a71c-70d6cf7da744/dns-node-resolver/0.log"
Sep 09 21:09:53.606980 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.606932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:09:53.607394 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.607371    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:09:53.762322 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.762281    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-97fdbc685-p5s4b_121425d6-de91-426a-bcee-c44d2adc5f24/registry/0.log"
Sep 09 21:09:53.790541 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:53.790506    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65pnk_8d10fd35-665c-41d1-9d16-56bb815fc8b6/node-ca/0.log"
Sep 09 21:09:55.141207 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:55.141174    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x28rk_d26c70cd-2e0d-4073-aeb0-5775d8d1983e/serve-healthcheck-canary/0.log"
Sep 09 21:09:55.545597 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:55.545523    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp476_a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf/kube-rbac-proxy/0.log"
Sep 09 21:09:55.567847 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:55.567821    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp476_a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf/exporter/0.log"
Sep 09 21:09:55.592254 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:55.592228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rp476_a3f3a2bb-d41d-4c1c-9ef2-3ca93b5b34cf/extractor/0.log"
Sep 09 21:09:58.265660 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.265626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w9tpv/perf-node-gather-daemonset-kwqzf"
Sep 09 21:09:58.327251 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.327215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-rjhx6_c51e2be1-75c5-4565-8c39-77250feceb42/server/0.log"
Sep 09 21:09:58.683987 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.683955    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-frblp_1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7/s3-init/0.log"
Sep 09 21:09:58.725948 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.725920    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-frblp_1150fc2e-5ef1-4b79-9fe1-f4c3c262deb7/download-hf-model/0.log"
Sep 09 21:09:58.791609 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.791581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7frmd_457db727-0f95-4d45-afe3-88f90b91a87b/s3-tls-init-custom/0.log"
Sep 09 21:09:58.830296 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.830259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7frmd_457db727-0f95-4d45-afe3-88f90b91a87b/download-hf-model/0.log"
Sep 09 21:09:58.864361 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.864331    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-t8th9_d5d10841-e2fa-4a6b-9ef3-94d951276da9/s3-tls-init-serving/0.log"
Sep 09 21:09:58.888375 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:58.888352    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-t8th9_d5d10841-e2fa-4a6b-9ef3-94d951276da9/download-hf-model/0.log"
Sep 09 21:09:59.000511 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:09:59.000426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-4rdx5_86e0e92a-3373-47e2-beb3-3aa3fb4830dc/seaweedfs-tls-serving/0.log"
Sep 09 21:10:03.607191 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:03.607152    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 21:10:03.607691 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:03.607665    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-6757b57d7d-mj5n5" podUID="6213c661-f8ab-48cd-982c-75851ecf0583" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:10:04.885567 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.885537    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:10:04.908340 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.908314    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/egress-router-binary-copy/0.log"
Sep 09 21:10:04.930655 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.930632    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/cni-plugins/0.log"
Sep 09 21:10:04.952793 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.952769    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/bond-cni-plugin/0.log"
Sep 09 21:10:04.974461 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.974438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/routeoverride-cni/0.log"
Sep 09 21:10:04.995802 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:04.995777    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/whereabouts-cni-bincopy/0.log"
Sep 09 21:10:05.017386 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:05.017358    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b2qwb_c99e68cf-f223-4525-83d4-2753c3a7035f/whereabouts-cni/0.log"
Sep 09 21:10:05.421535 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:05.421501    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sffbl_a13d5d69-cb99-4972-a456-f3d71e00d5fe/kube-multus/0.log"
Sep 09 21:10:05.446074 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:05.446049    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dsvjt_ffa78a3f-5d6e-4689-a436-65a2aafc1e01/network-metrics-daemon/0.log"
Sep 09 21:10:05.468257 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:05.468233    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dsvjt_ffa78a3f-5d6e-4689-a436-65a2aafc1e01/kube-rbac-proxy/0.log"
Sep 09 21:10:06.724036 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.724002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/ovn-controller/0.log"
Sep 09 21:10:06.740360 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.740333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/ovn-acl-logging/0.log"
Sep 09 21:10:06.765360 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.765331    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/kube-rbac-proxy-node/0.log"
Sep 09 21:10:06.787302 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.787271    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:10:06.808010 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.807978    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/northd/0.log"
Sep 09 21:10:06.826977 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.826957    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/nbdb/0.log"
Sep 09 21:10:06.848231 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.848205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/sbdb/0.log"
Sep 09 21:10:06.898589 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:06.898562    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z9m2s_3c9de731-69fd-49ab-8111-3b88aaef3bcf/ovnkube-controller/0.log"
Sep 09 21:10:07.882789 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:07.882759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2tw8s_8517e6d9-209b-4478-acaa-882e2d592fb8/network-check-target-container/0.log"
Sep 09 21:10:08.859059 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:08.858972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-t992m_43b73432-d77c-4330-b46f-4d25a13a610d/iptables-alerter/0.log"
Sep 09 21:10:09.480693 ip-10-0-129-45 kubenswrapper[2579]: I0909 21:10:09.480665    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9bjqp_872fbedf-f91b-469e-8cc7-f66d8d736696/tuned/0.log"