Sep 18 19:36:01.522587 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:36:01.522599 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:36:01.522609 ip-10-0-136-207 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:36:01.522916 ip-10-0-136-207 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:36:11.990941 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 18 19:36:11.990953 ip-10-0-136-207 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3e740657ad04af893dd67efc39eb771 --
Sep 18 19:38:09.925620 ip-10-0-136-207 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:13.574209 ip-10-0-136-207 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:13.574804 ip-10-0-136-207 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:13.574804 ip-10-0-136-207 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:13.574804 ip-10-0-136-207 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:13.574804 ip-10-0-136-207 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:13.574804 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.574323    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:13.577336 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577321    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.577336 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577335    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577339    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577342    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577346    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577348    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577351    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577354    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577357    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577359    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577362    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577364    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577367    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577370    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577372    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577375    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577377    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577380    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577383    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577386    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577393    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.577400 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577396    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577399    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577402    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577405    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577408    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577411    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577413    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577416    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577419    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577422    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577424    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577427    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577429    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577432    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577434    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577437    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577439    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577442    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577444    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577447    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.577882 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577449    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577453    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577457    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577460    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577463    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577465    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577467    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577470    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577473    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577475    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577477    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577480    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577483    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577485    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577489    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577491    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577495    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577497    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577500    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577503    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.578419 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577505    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577508    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577511    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577514    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577516    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577519    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577521    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577524    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577526    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577529    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577532    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577534    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577537    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577540    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577542    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577545    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577548    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577550    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577553    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577555    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.578918 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577558    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577560    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577563    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577565    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577568    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577571    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577574    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577579    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577582    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577587    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577590    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577593    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577596    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577599    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577602    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577604    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577607    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.577610    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.579429 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578278    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578286    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578291    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578294    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578297    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578300    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578303    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578306    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578309    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578311    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578314    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578317    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578319    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578322    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578325    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578327    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578330    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578332    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578335    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.579866 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578337    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578341    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578344    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578347    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578349    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578352    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578355    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578357    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578360    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578362    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578365    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578367    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578370    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578372    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578375    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578377    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578380    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578383    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578385    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578388    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.580392 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578390    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578393    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578395    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578398    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578400    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578403    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578406    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578408    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578411    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578413    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578416    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578419    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578421    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578425    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578428    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578432    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578435    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578439    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578441    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.580876 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578444    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578448    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578452    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578456    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578459    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578462    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578465    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578467    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578470    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578473    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578475    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578478    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578481    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578483    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578486    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578488    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578491    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578494    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578496    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578499    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.581364 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578502    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578504    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578507    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578509    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578512    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578514    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578519    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578521    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578525    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578527    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578530    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578532    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578535    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578538    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578540    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578543    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578545    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578548    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578550    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578553    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.581844 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578555    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578942    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578947    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578950    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578953    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578956    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578959    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578961    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578965    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578969    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578972    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578975    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578978    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578980    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578983    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578986    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578989    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578992    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578995    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.582341 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.578997    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579000    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579003    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579006    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579008    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579011    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579014    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579017    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579019    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579022    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579024    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579027    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579029    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579032    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579035    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579037    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579040    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579042    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579045    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579047    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.582854 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579051    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579054    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579056    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579059    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579061    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579064    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579066    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579069    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579071    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579074    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579077    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579080    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579083    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579085    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579088    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579092    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579096    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579110    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579113    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.583368 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579117    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579120    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579123    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579125    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579128    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579131    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579133    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579136    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579138    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579142    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579144    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579147    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579150    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579153    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579155    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579158    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579160    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579163    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579165    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579168    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.583836 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579170    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579173    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579176    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579178    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579182    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579185    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579187    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579190    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579193    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579196    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579214    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579217    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579220    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579222    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579225    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579227    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579230    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579233    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579235    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.584349 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579238    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579241    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579243    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579303    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579337    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579343    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579348    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579352    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579357    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579364    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579369    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579372    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579375    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579380    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579383    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579387    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579391    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579394    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579397    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579400    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579402    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579408    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579411    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:13.584812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579414    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579417    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579421    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579424    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579427    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579430    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579434    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579437    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579440    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579444    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579447    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579450    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579454    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579457    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579459    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579462    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579465    2573 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579469    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579474    2573 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579476    2573 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579479    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579482    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579485    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579489    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:13.585387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579494    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579497    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579500    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579503    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579506    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579508    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579511    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579514    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579517    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579520    2573 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579524    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579527    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579530    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579533    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579537    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579540    2573 flags.go:64] FLAG: --help="false"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579543    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579546    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579548    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579551    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579555    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579558    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579561    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:13.585966 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579564    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579566    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579570    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579573    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579576    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579579    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579582    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579585    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579588    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579592    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579595    2573 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579597    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579600    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579603    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579608    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579611    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579613    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579616    2573 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579619    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579623    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579626    2573 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579629    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579633    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579636    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579639    2573 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:13.586916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579642    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579645    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579648    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579651    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579654    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579657    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579660    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579667    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579670    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579674    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579677    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579680    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579685    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579688    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579691    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579694    2573 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579699    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579702    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e38b5f7b8eb1f760"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579705    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579708    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579711    2573 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579714    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579717    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579720    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:13.587658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579723    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579726    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579730    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579733    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579736    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579739    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579742    2573 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579745    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579748    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579751    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579753    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579756    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579759    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579762    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579765    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579768    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579771    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579774    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579778    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579781    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579784    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579786    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579791    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579794    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579798    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:13.588246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579803    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579806    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579808    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579811    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579814    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579817    2573 flags.go:64] FLAG: --v="2"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579821    2573 flags.go:64] FLAG: --version="false"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579825    2573 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579828    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.579832    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579948    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579952    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579955    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579958    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579960    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579963    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579965    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579968    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579971    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579973    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579976    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.588861 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579979    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579981    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579984    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579986    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579990    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579993    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579996    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.579998    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580001    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580003    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580007    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580009    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580012    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580014    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580017    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580019    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580022    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580025    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580027    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580030    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.589391 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580032    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580035    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580038    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580041    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580043    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580046    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580048    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580053    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580056    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580059    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580062    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580065    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580068    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580071    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580074    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580077    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580081    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580084    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580086    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580089    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.589888 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580092    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580094    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580097    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580100    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580103    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580105    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580108    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580110    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580113    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580115    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580118    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580122    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580125    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580128    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580131    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580135    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580137    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580140    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580143    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.590417 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580145    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580149    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580153    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580155    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580158    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580160    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580163    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580165    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580168    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580171    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580174    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580177    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580179    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580182    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580184    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580187    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580190    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580193    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580195    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580216    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.590897 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580219    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580222    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580224    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580227    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580230    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580233    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580235    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580238    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.580241    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.580743    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589222    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589237    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589258    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589264    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:13.591418 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589319    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589323    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589327    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589331    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589334    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589337    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589340    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589343    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589346    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589349    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589351    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589354    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589357    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589359    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589362    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589365    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589367    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589370    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589373    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589376    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.591774 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589378    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589381    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589383    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589386    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589388    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589391    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589394    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589397    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589400    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589403    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589406    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589409    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589412    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589415    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589417    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589420    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589423    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589426    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589428    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589431    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.592359 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589434    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589437    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589439    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589442    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589444    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589447    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589450    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589452    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589455    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589458    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589460    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589463    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589465    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589468    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589471    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589473    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589476    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589479    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589482    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589484    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.592859 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589487    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589489    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589492    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589495    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589498    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589500    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589503    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589506    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589508    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589512    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589516    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589519    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589522    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589524    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589528    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589532    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589535    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589538    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589540    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.593386 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589544    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589546    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589549    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589551    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589554    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589557    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589560    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589562    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589565    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589568    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589570    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589573    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589577    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589580    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589583    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589586    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589589    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589591    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589594    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.593848 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589597    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589602    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589715    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589720    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589723    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589726    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589728    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589731    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589734    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589738    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589742    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589745    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589748    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589751    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:13.594332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589755    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589758    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589761    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589763    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589766    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589768    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589771    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589774    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589777    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589779    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589782    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589784    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589787    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589790    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589792    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589795    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589798    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589801    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589803    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589806    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:13.594683 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589808    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589811    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589813    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589816    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589818    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589821    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589823    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589826    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589828    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589831    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589833    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589836    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589839    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589842    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589844    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589847    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589849    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589852    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589855    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589858    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:13.595182 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589860    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589863    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589866    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589868    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589871    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589873    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589876    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589878    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589881    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589885    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589888    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589891    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589894    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589897    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589900    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589903    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589906    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589909    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589911    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589914    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:13.595692 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589916    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589919    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589922    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589924    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589927    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589929    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589936    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589939    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589942    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589944    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589947    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589950    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589953    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589955    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589958    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589963    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589965    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589968    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589970    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589973    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:13.596229 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589976    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589978    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589981    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589984    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589986    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589990    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:13.589993    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.589998    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.590090    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.592039    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.593059    2573 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.593152    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.594144    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:13.596718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.595854    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:13.597648 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.597630    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:13.609269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.609250    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:13.611701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.611680    2573 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:13.613116 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.613097    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:13.615334 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.615313    2573 fs.go:135] Filesystem UUIDs: map[236099ae-3c5d-466d-8c29-688372d902e8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b61858f-c1e9-404a-9fe1-4f4bcb94412b:/dev/nvme0n1p4]
Sep 18 19:38:13.615403 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.615330    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:13.620318 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.620211    2573 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:13.619747219 +0000 UTC m=+2.137716851 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104118 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e663b042ad0a6c50bcbe69ff1ada5 SystemUUID:ec2e663b-042a-d0a6-c50b-cbe69ff1ada5 BootID:b3e74065-7ad0-4af8-93dd-67efc39eb771 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:39:67:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:39:67:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:c2:64:63:04:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:13.620318 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.620309    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:13.620458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.620399    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:13.621842 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.621815    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:13.621994 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.621844    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:13.623018 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623005    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:13.623077 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623023    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:13.623077 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623034    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:13.623077 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623049    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:13.623274 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623128    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:13.623365 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.623349    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:13.627461 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627445    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:13.627590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627579    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:13.627645 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627617    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:13.627645 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627631    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:13.627744 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627651    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:13.627744 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627665    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:13.627744 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627676    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:13.628010 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.627996    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:13.628069 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.628020    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:13.630020 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.629999    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:13.632373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.632359    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:13.632447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.632388    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:13.633366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633353    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633369    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633375    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633380    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633389    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633394    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633408    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:13.633413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633413    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633420    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633426    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633434    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633439    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633461    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:13.633618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.633467    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:13.637455 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.637441    2573 server.go:1268] "Started kubelet"
Sep 18 19:38:13.637560 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.637518    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:13.637666 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.637588    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:13.637666 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.637594    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:13.638253 ip-10-0-136-207 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:13.638716 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.638666    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:13.639601 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.639499    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:13.669110 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.669078    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:13.669220 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.669158    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:13.669496 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.669471    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:13.670238 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.670164    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:13.670676 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.670641    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kbqv6"
Sep 18 19:38:13.670676 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.670676    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kbqv6"
Sep 18 19:38:13.674121 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674101    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:13.674228 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674106    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:13.674323 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.674264    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:13.674652 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674626    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:13.674744 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674724    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:13.674814 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674746    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:13.674869 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674845    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:13.674869 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.674854    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:13.675393 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.674483    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-207.ec2.internal.18d68181275b56f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-207.ec2.internal,UID:ip-10-0-136-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-207.ec2.internal,},FirstTimestamp:2026-09-18 19:38:13.637412594 +0000 UTC m=+2.155382224,LastTimestamp:2026-09-18 19:38:13.637412594 +0000 UTC m=+2.155382224,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-207.ec2.internal,}"
Sep 18 19:38:13.675540 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675529    2573 factory.go:55] Registering systemd factory
Sep 18 19:38:13.675578 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675542    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:13.675777 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675762    2573 factory.go:153] Registering CRI-O factory
Sep 18 19:38:13.675829 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675781    2573 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:13.675859 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675833    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:13.675859 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675851    2573 factory.go:103] Registering Raw factory
Sep 18 19:38:13.675923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.675864    2573 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:13.676189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.676175    2573 manager.go:349] Starting recovery of all containers
Sep 18 19:38:13.685411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.685392    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:13.686698 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.686681    2573 manager.go:354] Recovery completed
Sep 18 19:38:13.690999 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.690983    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.691923 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.691908    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.692599 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.692586    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.692655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.692612    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.692655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.692622    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.693142 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693129    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:13.693142 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693141    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:13.693272 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693150    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.693272 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693155    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:13.693871 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693854    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.693940 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693885    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.693940 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.693894    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.695169 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.695155    2573 policy_none.go:49] "None policy: Start"
Sep 18 19:38:13.695169 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.695171    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:13.695280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.695180    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:13.697120 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.697108    2573 policy_none.go:47] "Start"
Sep 18 19:38:13.734768 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.734754    2573 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.734803    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.734815    2573 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.735044    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.735087    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.735188    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.735297    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.735306    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.735833    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:13.746597 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.735871    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:13.775559 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.775537    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:13.776666 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.776647    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:13.776746 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.776699    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:13.776746 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.776723    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:13.776837 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.776764    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:13.778939 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.778921    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:13.835605 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.835562    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.836393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.836379    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.836449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.836407    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.836449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.836416    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.836449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.836439    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.844001 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.843987    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.844049 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.844007    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-207.ec2.internal\": node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:13.861276 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.861258    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:13.877553 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.877533    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"]
Sep 18 19:38:13.877614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.877586    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.878839 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.878818    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.878906 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.878843    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.878906 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.878852    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.880984 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.880973    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.881105 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.881139 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881120    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.881644 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881627    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.881644 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881637    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.881742 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881660    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.881742 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881674    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.881742 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881731    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.881847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.881752    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.883789 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.883775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.883840 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.883806    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:13.884496 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.884480    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:13.884564 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.884509    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:13.884564 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:13.884523    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:13.896800 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.896781    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.900060 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.900045    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal"
Sep 18 19:38:13.961632 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:13.961614    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.062468 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.062441    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.076804 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.076788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17a176fdef57f822539a4bd26e5f2da2-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"17a176fdef57f822539a4bd26e5f2da2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.076855 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.076840    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.076889 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.076859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.163237 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.163169    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.177495 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.177544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17a176fdef57f822539a4bd26e5f2da2-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"17a176fdef57f822539a4bd26e5f2da2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.177544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.177616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.177616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f19f41f7a65d19182efb2f9c1106ad56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"f19f41f7a65d19182efb2f9c1106ad56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.177616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.177588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17a176fdef57f822539a4bd26e5f2da2-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"17a176fdef57f822539a4bd26e5f2da2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.200607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.200591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.203755 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.203741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.263904 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.263875    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.364347 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.364326    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.464893 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.464843    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found"
Sep 18 19:38:14.563064 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.563041    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:14.574886 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.574867    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.586373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.586357    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:14.588367 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.588355    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.592499 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.592486    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:14.592594 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.592580    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:14.592648 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.592613    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:14.592648 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.592621    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:14.592710 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.592642    2573 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a53a7ad50fef7478eaba743342f0d1b3-94cb3a4bdf0471de.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.136.207:37986->100.59.172.159:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:14.628794 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.628775    2573 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:14.642159 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.642019    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:14.642385 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.642366    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal","openshift-multus/multus-additional-cni-plugins-kx2s7","openshift-multus/multus-xcshw","kube-system/konnectivity-agent-fwbh5","openshift-multus/network-metrics-daemon-8grvn","openshift-network-diagnostics/network-check-target-kh7cd","openshift-network-operator/iptables-alerter-wmlkl","openshift-ovn-kubernetes/ovnkube-node-ds2kn","kube-system/global-pull-secret-syncer-f75sh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5","openshift-cluster-node-tuning-operator/tuned-rwjqg","openshift-dns/node-resolver-29vx8","openshift-image-registry/node-ca-7cxk6"]
Sep 18 19:38:14.644711 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.644694    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.646856 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.646841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.647743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.647724    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:14.647743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.647728    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.647879 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.647776    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.647983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.647970    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lkkh6\""
Sep 18 19:38:14.649037 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.649005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4v2kz\""
Sep 18 19:38:14.649272 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.649254    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.650162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.649816    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:14.650162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.649831    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:14.650162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.650018    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:14.650364 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.650339    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.650438 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.650420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.651152 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.650664    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:14.652608 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.652579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.652712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.652627    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f79x2\""
Sep 18 19:38:14.652887 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.652873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:14.654736 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.654719    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:14.654820 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.654742    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:14.654956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.654942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8vdd7\""
Sep 18 19:38:14.655551 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.655536    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.655648 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.655612    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:14.657612 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.657591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:14.657709 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.657650    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:14.659819 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.659801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.662081 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662064    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.662170 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.662125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:14.662340 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-njf2c\""
Sep 18 19:38:14.662393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662371    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:14.662428 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662407    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:14.662428 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662415    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:14.662485 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662445    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.662519 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662510    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:14.662652 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.662640    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.664556 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.664539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.666787 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.666768    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:14.666896 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.666788    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wpglg\""
Sep 18 19:38:14.666896 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.666770    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.666896 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.666867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.667053 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.666927    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.669174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.669157    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.669294 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.669184    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8wxc8\""
Sep 18 19:38:14.669396 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.669376    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.669678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.669661    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.671577 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.671561    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-75q56\""
Sep 18 19:38:14.671889 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.671873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.671956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.671873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.672446 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.672430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.674269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.674251    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:14.674392 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.674369    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:14.674850 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.674832    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n56mj\""
Sep 18 19:38:14.674950 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.674859    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:14.675111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.675099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:14.675499 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.675486    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:14.675633 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.675568    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:13 +0000 UTC" deadline="2028-05-13 13:28:27.588955984 +0000 UTC"
Sep 18 19:38:14.675685 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.675635    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14465h50m12.913325327s"
Sep 18 19:38:14.680410 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-systemd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-log-socket\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680613 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a37a033-426d-4229-bebf-67a87e8d980a-host-slash\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.680613 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-kubelet-config\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.680613 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwv7\" (UniqueName: \"kubernetes.io/projected/87591871-d05d-4e65-ba24-915d085bf6b2-kube-api-access-hvwv7\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.680723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-kubelet\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.680723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-hostroot\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.680723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.680723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-lib-modules\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-host\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-script-lib\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680821    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cni-binary-copy\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-k8s-cni-cncf-io\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-registration-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.680921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-system-cni-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-kubelet\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.680986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-dbus\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-device-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-sys-fs\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-tmp-dir\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.681447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-multus-certs\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-kubernetes\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-conf\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmtm\" (UniqueName: \"kubernetes.io/projected/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-kube-api-access-prmtm\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-var-lib-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-netd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-config\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.681779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-bin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-etc-kubernetes\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e010f81-8cd6-4365-83df-7651a560fba1-agent-certs\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.681977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovn-node-metrics-cert\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-run\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.682128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-tmp\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-hosts-file\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-node-log\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a37a033-426d-4229-bebf-67a87e8d980a-iptables-alerter-script\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-var-lib-kubelet\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.682429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g42qf\" (UniqueName: \"kubernetes.io/projected/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-kube-api-access-g42qf\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-conf-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq777\" (UniqueName: \"kubernetes.io/projected/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-kube-api-access-cq777\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-slash\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-ovn\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-env-overrides\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-netns\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-multus\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-socket-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-tuned\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-systemd-units\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.682956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.683218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.682974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.683218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8qr\" (UniqueName: \"kubernetes.io/projected/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kube-api-access-5x8qr\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.683218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-etc-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.683218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683102    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0385842a-4021-45f6-9d3d-606a34d90b5e-host\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.683218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslkz\" (UniqueName: \"kubernetes.io/projected/0385842a-4021-45f6-9d3d-606a34d90b5e-kube-api-access-wslkz\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.683458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-socket-dir-parent\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.683458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-daemon-config\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.683458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-netns\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.683458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tt65\" (UniqueName: \"kubernetes.io/projected/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-kube-api-access-5tt65\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.683458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cnibin\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57zbl\" (UniqueName: \"kubernetes.io/projected/3a37a033-426d-4229-bebf-67a87e8d980a-kube-api-access-57zbl\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683537    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-system-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cnibin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-modprobe-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.683701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e010f81-8cd6-4365-83df-7651a560fba1-konnectivity-ca\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysconfig\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-systemd\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-sys\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-bin\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7t5n\" (UniqueName: \"kubernetes.io/projected/34becf8e-bd90-4e82-80c0-ff527bee92bd-kube-api-access-f7t5n\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.684022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.683980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0385842a-4021-45f6-9d3d-606a34d90b5e-serviceca\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.684885 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.684868    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:14.712213 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.712187    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t48cr"
Sep 18 19:38:14.720916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.720903    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t48cr"
Sep 18 19:38:14.784628 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-systemd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-log-socket\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a37a033-426d-4229-bebf-67a87e8d980a-host-slash\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-kubelet-config\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.784730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784723    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-log-socket\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784723    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784725    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-systemd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a37a033-426d-4229-bebf-67a87e8d980a-host-slash\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784778    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-kubelet-config\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784792    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwv7\" (UniqueName: \"kubernetes.io/projected/87591871-d05d-4e65-ba24-915d085bf6b2-kube-api-access-hvwv7\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-kubelet\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-hostroot\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784885    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-kubelet\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784903    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-hostroot\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.784930    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.784981    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:15.284962844 +0000 UTC m=+3.802932466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:14.784991 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.784999    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-lib-modules\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-host\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-script-lib\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-etc-selinux\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cni-binary-copy\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785093    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-host\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-k8s-cni-cncf-io\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785110    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-lib-modules\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-registration-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785167    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-k8s-cni-cncf-io\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-system-cni-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-kubelet\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785243    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-system-cni-dir\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-dbus\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-device-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.785827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-sys-fs\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785306    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-registration-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-sys-fs\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-device-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-kubelet\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-dbus\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-tmp-dir\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-multus-certs\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-multus-certs\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-kubernetes\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-conf\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmtm\" (UniqueName: \"kubernetes.io/projected/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-kube-api-access-prmtm\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-var-lib-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785590    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-netd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-config\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-kubernetes\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.786718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-bin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-var-lib-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cni-binary-copy\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-etc-kubernetes\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e010f81-8cd6-4365-83df-7651a560fba1-agent-certs\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785724    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-bin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysctl-conf\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-script-lib\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785755    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-etc-kubernetes\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-netd\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-tmp-dir\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovn-node-metrics-cert\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-run\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-tmp\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-hosts-file\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.787571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.785974    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-hosts-file\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-node-log\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a37a033-426d-4229-bebf-67a87e8d980a-iptables-alerter-script\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-var-lib-kubelet\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786043    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-run\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g42qf\" (UniqueName: \"kubernetes.io/projected/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-kube-api-access-g42qf\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-conf-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovnkube-config\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq777\" (UniqueName: \"kubernetes.io/projected/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-kube-api-access-cq777\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-node-log\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-slash\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-var-lib-kubelet\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786296    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-slash\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-conf-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-ovn\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.788408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786364    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-env-overrides\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-netns\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-multus\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-run-ovn\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-socket-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-tuned\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786519    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-socket-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-systemd-units\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-var-lib-cni-multus\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8qr\" (UniqueName: \"kubernetes.io/projected/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kube-api-access-5x8qr\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-etc-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0385842a-4021-45f6-9d3d-606a34d90b5e-host\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslkz\" (UniqueName: \"kubernetes.io/projected/0385842a-4021-45f6-9d3d-606a34d90b5e-kube-api-access-wslkz\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-socket-dir-parent\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.791171 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-daemon-config\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-netns\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tt65\" (UniqueName: \"kubernetes.io/projected/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-kube-api-access-5tt65\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/34becf8e-bd90-4e82-80c0-ff527bee92bd-env-overrides\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786803    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cnibin\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786809    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3a37a033-426d-4229-bebf-67a87e8d980a-iptables-alerter-script\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786840    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-systemd-units\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786846    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-cnibin\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786929    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-host-run-netns\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-etc-openvswitch\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.787091    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.787146    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:15.287128934 +0000 UTC m=+3.805098567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-socket-dir-parent\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.786873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57zbl\" (UniqueName: \"kubernetes.io/projected/3a37a033-426d-4229-bebf-67a87e8d980a-kube-api-access-57zbl\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.792192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-system-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-run-netns\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cnibin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-modprobe-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787314    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e010f81-8cd6-4365-83df-7651a560fba1-konnectivity-ca\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-system-cni-dir\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysconfig\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787419    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-cnibin\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-systemd\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-modprobe-d\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.787651    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0385842a-4021-45f6-9d3d-606a34d90b5e-host\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.788175    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-systemd\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.788310    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-sys\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.788330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-sys\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.788571    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-multus-daemon-config\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.788957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e010f81-8cd6-4365-83df-7651a560fba1-konnectivity-ca\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.792898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-sysconfig\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-bin\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789189    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7t5n\" (UniqueName: \"kubernetes.io/projected/34becf8e-bd90-4e82-80c0-ff527bee92bd-kube-api-access-f7t5n\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34becf8e-bd90-4e82-80c0-ff527bee92bd-host-cni-bin\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0385842a-4021-45f6-9d3d-606a34d90b5e-serviceca\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0385842a-4021-45f6-9d3d-606a34d90b5e-serviceca\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.789861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-etc-tuned\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.791083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/34becf8e-bd90-4e82-80c0-ff527bee92bd-ovn-node-metrics-cert\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.793554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.793073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e010f81-8cd6-4365-83df-7651a560fba1-agent-certs\") pod \"konnectivity-agent-fwbh5\" (UID: \"6e010f81-8cd6-4365-83df-7651a560fba1\") " pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:14.793992 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.793752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-tmp\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.795152 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.795134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmtm\" (UniqueName: \"kubernetes.io/projected/c46ec78c-24d5-4a08-9c45-f5b3cebc62dc-kube-api-access-prmtm\") pod \"tuned-rwjqg\" (UID: \"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc\") " pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:14.795458 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.795442    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwv7\" (UniqueName: \"kubernetes.io/projected/87591871-d05d-4e65-ba24-915d085bf6b2-kube-api-access-hvwv7\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:14.797291 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.797275    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:14.797291 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.797292    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:14.797408 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.797301    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:14.797408 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:14.797342    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:15.297331539 +0000 UTC m=+3.815301157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:14.799516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.799478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8qr\" (UniqueName: \"kubernetes.io/projected/6dae77fb-13e9-43cb-9e44-4ce5dce5fab0-kube-api-access-5x8qr\") pod \"aws-ebs-csi-driver-node-4p7p5\" (UID: \"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:14.800184 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.800159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslkz\" (UniqueName: \"kubernetes.io/projected/0385842a-4021-45f6-9d3d-606a34d90b5e-kube-api-access-wslkz\") pod \"node-ca-7cxk6\" (UID: \"0385842a-4021-45f6-9d3d-606a34d90b5e\") " pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.800766 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.800742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g42qf\" (UniqueName: \"kubernetes.io/projected/4e1f3ce1-13e9-4c68-afca-dc5e17918cac-kube-api-access-g42qf\") pod \"multus-additional-cni-plugins-kx2s7\" (UID: \"4e1f3ce1-13e9-4c68-afca-dc5e17918cac\") " pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.800882 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.800862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57zbl\" (UniqueName: \"kubernetes.io/projected/3a37a033-426d-4229-bebf-67a87e8d980a-kube-api-access-57zbl\") pod \"iptables-alerter-wmlkl\" (UID: \"3a37a033-426d-4229-bebf-67a87e8d980a\") " pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.800998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.800974    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tt65\" (UniqueName: \"kubernetes.io/projected/fd7438f0-23d8-42bd-82fc-fd43c3f98c1b-kube-api-access-5tt65\") pod \"node-resolver-29vx8\" (UID: \"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b\") " pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:14.801708 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.801690    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq777\" (UniqueName: \"kubernetes.io/projected/44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c-kube-api-access-cq777\") pod \"multus-xcshw\" (UID: \"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c\") " pod="openshift-multus/multus-xcshw"
Sep 18 19:38:14.802984 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.802967    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7t5n\" (UniqueName: \"kubernetes.io/projected/34becf8e-bd90-4e82-80c0-ff527bee92bd-kube-api-access-f7t5n\") pod \"ovnkube-node-ds2kn\" (UID: \"34becf8e-bd90-4e82-80c0-ff527bee92bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:14.805444 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.805427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cxk6"
Sep 18 19:38:14.899097 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:14.899060    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19f41f7a65d19182efb2f9c1106ad56.slice/crio-46180de32adaab4400d559f4b8e770f59b3d8d1158b21bd3103cc618575b32a3 WatchSource:0}: Error finding container 46180de32adaab4400d559f4b8e770f59b3d8d1158b21bd3103cc618575b32a3: Status 404 returned error can't find the container with id 46180de32adaab4400d559f4b8e770f59b3d8d1158b21bd3103cc618575b32a3
Sep 18 19:38:14.900024 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:14.900003    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a176fdef57f822539a4bd26e5f2da2.slice/crio-42166e05f18533b3a41d21d03fa9af4e0252a135f2011e54e0344e0997ec2469 WatchSource:0}: Error finding container 42166e05f18533b3a41d21d03fa9af4e0252a135f2011e54e0344e0997ec2469: Status 404 returned error can't find the container with id 42166e05f18533b3a41d21d03fa9af4e0252a135f2011e54e0344e0997ec2469
Sep 18 19:38:14.904069 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.904056    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:14.981738 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.981672    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wmlkl"
Sep 18 19:38:14.987745 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:14.987728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx2s7"
Sep 18 19:38:14.991022 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:14.990997    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a37a033_426d_4229_bebf_67a87e8d980a.slice/crio-2ceadf9eb0e89d4090118169fa8b1fa708ee798233a1d59d29355ebdf7f4aa40 WatchSource:0}: Error finding container 2ceadf9eb0e89d4090118169fa8b1fa708ee798233a1d59d29355ebdf7f4aa40: Status 404 returned error can't find the container with id 2ceadf9eb0e89d4090118169fa8b1fa708ee798233a1d59d29355ebdf7f4aa40
Sep 18 19:38:14.995607 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:14.995584    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1f3ce1_13e9_4c68_afca_dc5e17918cac.slice/crio-d0d23cae63467cebb38c349588ed18ca120812c8369e193fcd8e96a66235d951 WatchSource:0}: Error finding container d0d23cae63467cebb38c349588ed18ca120812c8369e193fcd8e96a66235d951: Status 404 returned error can't find the container with id d0d23cae63467cebb38c349588ed18ca120812c8369e193fcd8e96a66235d951
Sep 18 19:38:15.006976 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.006959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcshw"
Sep 18 19:38:15.015940 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.015912    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a30c5a_0fc4_4da3_bc2f_8a67d96fc74c.slice/crio-5613fd880e458fe0cde8989d0dc0690954a4944ae1a280936b2efe61e1e880c6 WatchSource:0}: Error finding container 5613fd880e458fe0cde8989d0dc0690954a4944ae1a280936b2efe61e1e880c6: Status 404 returned error can't find the container with id 5613fd880e458fe0cde8989d0dc0690954a4944ae1a280936b2efe61e1e880c6
Sep 18 19:38:15.018607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.018531    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:15.024917 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.024902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:15.028252 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.028229    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e010f81_8cd6_4365_83df_7651a560fba1.slice/crio-f1ac996db7fe1ba022c7f90b7fe91dd7253205232703331dad9eaff650eaf965 WatchSource:0}: Error finding container f1ac996db7fe1ba022c7f90b7fe91dd7253205232703331dad9eaff650eaf965: Status 404 returned error can't find the container with id f1ac996db7fe1ba022c7f90b7fe91dd7253205232703331dad9eaff650eaf965
Sep 18 19:38:15.032531 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.032508    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34becf8e_bd90_4e82_80c0_ff527bee92bd.slice/crio-08a6ecaa5564983c3b305cdc74e88b4a4b549c479eb65c121196943e5ecbc63f WatchSource:0}: Error finding container 08a6ecaa5564983c3b305cdc74e88b4a4b549c479eb65c121196943e5ecbc63f: Status 404 returned error can't find the container with id 08a6ecaa5564983c3b305cdc74e88b4a4b549c479eb65c121196943e5ecbc63f
Sep 18 19:38:15.052080 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.052059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5"
Sep 18 19:38:15.058770 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.058750    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwjqg"
Sep 18 19:38:15.060413 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.060394    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dae77fb_13e9_43cb_9e44_4ce5dce5fab0.slice/crio-201f1b19dd4d84a46a56ad7105320c97be7fe864e598afc497803a50138a4647 WatchSource:0}: Error finding container 201f1b19dd4d84a46a56ad7105320c97be7fe864e598afc497803a50138a4647: Status 404 returned error can't find the container with id 201f1b19dd4d84a46a56ad7105320c97be7fe864e598afc497803a50138a4647
Sep 18 19:38:15.066273 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.066255    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46ec78c_24d5_4a08_9c45_f5b3cebc62dc.slice/crio-c98252b7497ba19be0a767681cfb8fc0a95f4cfc7a9aefe499819a185bb580b2 WatchSource:0}: Error finding container c98252b7497ba19be0a767681cfb8fc0a95f4cfc7a9aefe499819a185bb580b2: Status 404 returned error can't find the container with id c98252b7497ba19be0a767681cfb8fc0a95f4cfc7a9aefe499819a185bb580b2
Sep 18 19:38:15.090601 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.090584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-29vx8"
Sep 18 19:38:15.099268 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.099244    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7438f0_23d8_42bd_82fc_fd43c3f98c1b.slice/crio-1358642dedfe4cb6872464f16d58d19804397075cc39c78d33d83000975262cd WatchSource:0}: Error finding container 1358642dedfe4cb6872464f16d58d19804397075cc39c78d33d83000975262cd: Status 404 returned error can't find the container with id 1358642dedfe4cb6872464f16d58d19804397075cc39c78d33d83000975262cd
Sep 18 19:38:15.149043 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:15.149025    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0385842a_4021_45f6_9d3d_606a34d90b5e.slice/crio-bcb3ff285e0775690afba02d7c122074bd135dc286a74bc28f82e32db1fff708 WatchSource:0}: Error finding container bcb3ff285e0775690afba02d7c122074bd135dc286a74bc28f82e32db1fff708: Status 404 returned error can't find the container with id bcb3ff285e0775690afba02d7c122074bd135dc286a74bc28f82e32db1fff708
Sep 18 19:38:15.291940 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.291856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:15.291940 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.291922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:15.292181 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.292042    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.292181 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.292095    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.292181 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.292096    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.292079601 +0000 UTC m=+4.810049236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.292181 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.292139    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.292128746 +0000 UTC m=+4.810098379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.393027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.392991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:15.393230 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.393177    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:15.393230 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.393196    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:15.393230 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.393226    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.393375 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.393281    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:16.393263093 +0000 UTC m=+4.911232727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.724462 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.724334    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:14 +0000 UTC" deadline="2028-05-21 09:01:42.187959556 +0000 UTC"
Sep 18 19:38:15.724462 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.724368    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14653h23m26.463595361s"
Sep 18 19:38:15.779040 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.778928    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:15.779421 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:15.779380    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:15.813891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.813819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcshw" event={"ID":"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c","Type":"ContainerStarted","Data":"5613fd880e458fe0cde8989d0dc0690954a4944ae1a280936b2efe61e1e880c6"}
Sep 18 19:38:15.836165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.836127    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerStarted","Data":"d0d23cae63467cebb38c349588ed18ca120812c8369e193fcd8e96a66235d951"}
Sep 18 19:38:15.849656 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.849621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmlkl" event={"ID":"3a37a033-426d-4229-bebf-67a87e8d980a","Type":"ContainerStarted","Data":"2ceadf9eb0e89d4090118169fa8b1fa708ee798233a1d59d29355ebdf7f4aa40"}
Sep 18 19:38:15.863162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.863087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" event={"ID":"17a176fdef57f822539a4bd26e5f2da2","Type":"ContainerStarted","Data":"42166e05f18533b3a41d21d03fa9af4e0252a135f2011e54e0344e0997ec2469"}
Sep 18 19:38:15.880238 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.880193    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"f19f41f7a65d19182efb2f9c1106ad56","Type":"ContainerStarted","Data":"46180de32adaab4400d559f4b8e770f59b3d8d1158b21bd3103cc618575b32a3"}
Sep 18 19:38:15.901567 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.901470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cxk6" event={"ID":"0385842a-4021-45f6-9d3d-606a34d90b5e","Type":"ContainerStarted","Data":"bcb3ff285e0775690afba02d7c122074bd135dc286a74bc28f82e32db1fff708"}
Sep 18 19:38:15.933049 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.932963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29vx8" event={"ID":"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b","Type":"ContainerStarted","Data":"1358642dedfe4cb6872464f16d58d19804397075cc39c78d33d83000975262cd"}
Sep 18 19:38:15.949914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.949838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwjqg" event={"ID":"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc","Type":"ContainerStarted","Data":"c98252b7497ba19be0a767681cfb8fc0a95f4cfc7a9aefe499819a185bb580b2"}
Sep 18 19:38:15.956242 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.956215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5" event={"ID":"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0","Type":"ContainerStarted","Data":"201f1b19dd4d84a46a56ad7105320c97be7fe864e598afc497803a50138a4647"}
Sep 18 19:38:15.972340 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:15.972310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"08a6ecaa5564983c3b305cdc74e88b4a4b549c479eb65c121196943e5ecbc63f"}
Sep 18 19:38:16.012286 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.012213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fwbh5" event={"ID":"6e010f81-8cd6-4365-83df-7651a560fba1","Type":"ContainerStarted","Data":"f1ac996db7fe1ba022c7f90b7fe91dd7253205232703331dad9eaff650eaf965"}
Sep 18 19:38:16.141163 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.141138    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:16.194560 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.194498    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.301869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.301927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.302076    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.302091    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.302148    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.302127437 +0000 UTC m=+6.820097072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:16.302280 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.302166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.302156946 +0000 UTC m=+6.820126567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:16.403026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.402438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:16.403026 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.402589    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:16.403026 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.402608    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:16.403026 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.402620    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:16.403026 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.402673    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:18.402655696 +0000 UTC m=+6.920625334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:16.725441 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.725311    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:14 +0000 UTC" deadline="2028-05-26 10:00:19.188827674 +0000 UTC"
Sep 18 19:38:16.725441 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.725367    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14774h22m2.463466249s"
Sep 18 19:38:16.778674 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.777711    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:16.778674 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.777848    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:16.778674 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:16.778266    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:16.778674 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:16.778359    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:17.667395 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:17.667367    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:17.780600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:17.779869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:17.780600 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:17.780000    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:18.319640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:18.319707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.319842    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.319906    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.319887815 +0000 UTC m=+10.837857453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.320320    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:18.320452 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.320378    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.320361498 +0000 UTC m=+10.838331120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:18.421492 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:18.421009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:18.421492 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.421142    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:18.421492 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.421160    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:18.421492 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.421169    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:18.421492 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.421227    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:22.421212423 +0000 UTC m=+10.939182055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:18.777632 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:18.776901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:18.777632 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.777047    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:18.777632 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:18.777444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:18.777632 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:18.777537    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:19.777480 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:19.777448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:19.777941 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:19.777585    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:20.777983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:20.777707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:20.777983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:20.777755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:20.777983 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:20.777859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:20.777983 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:20.777953    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:21.777658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:21.777529    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:21.777822 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:21.777673    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:22.354692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:22.354652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:22.355164 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:22.354752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:22.355164 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.354805    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:22.355164 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.354861    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:22.355164 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.354876    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.354856694 +0000 UTC m=+18.872826326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:22.355164 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.354914    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.354898402 +0000 UTC m=+18.872868028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:22.455504 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:22.455468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:22.455684 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.455670    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:22.455755 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.455688    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:22.455755 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.455700    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:22.455755 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.455755    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:30.45573831 +0000 UTC m=+18.973707934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:22.777314 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:22.777237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:22.777468 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.777367    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:22.777600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:22.777237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:22.777712 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:22.777682    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:23.778604 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:23.778575    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:23.779050 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:23.778993    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:24.777815 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:24.777789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:24.777993 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:24.777917    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:24.777993 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:24.777982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:24.778114 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:24.778097    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:25.777393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:25.777358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:25.777825 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:25.777490    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:26.777414 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:26.777380    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:26.777900 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:26.777384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:26.777900 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:26.777517    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:26.777900 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:26.777588    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:27.777646 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:27.777616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:27.778126 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:27.777733    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:28.777981 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:28.777946    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:28.778436 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:28.778072    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:28.778436 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:28.778106    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:28.778436 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:28.778160    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:29.777570 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:29.777540    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:29.777737 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:29.777639    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:30.404848 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:30.404808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:30.405307 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:30.404902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:30.405307 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.404955    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.405307 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.405034    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.405012755 +0000 UTC m=+34.922982384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:30.405307 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.405035    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.405307 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.405094    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.405076681 +0000 UTC m=+34.923046318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:30.506165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:30.506132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:30.506372 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.506289    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:30.506372 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.506315    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:30.506372 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.506328    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.506532 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.506390    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:46.506373817 +0000 UTC m=+35.024343449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:30.777009 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:30.776937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:30.777155 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.777049    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:30.777155 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:30.777080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:30.777155 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:30.777146    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:31.777023 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:31.776991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:31.777546 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:31.777150    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:32.777270 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:32.777241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:32.777710 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:32.777369    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:32.777710 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:32.777426    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:32.777710 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:32.777549    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:33.777600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:33.777453    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:33.778039 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:33.777674    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:34.059010 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.058761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcshw" event={"ID":"44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c","Type":"ContainerStarted","Data":"88f7d8a87c80c68e6d4dc1ab221a3d9e8295f580a70561b1595309278479e41f"}
Sep 18 19:38:34.060827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.060777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" event={"ID":"17a176fdef57f822539a4bd26e5f2da2","Type":"ContainerStarted","Data":"123056b2d465607f8fd9debb33ac74e2817ca8a8d034a95800d91e89c614adb1"}
Sep 18 19:38:34.060827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.060795    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:34.066400 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.066361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwjqg" event={"ID":"c46ec78c-24d5-4a08-9c45-f5b3cebc62dc","Type":"ContainerStarted","Data":"8823c07c8d6422f2413392c70d06c91a6b443fdc849169d9d95b637ab2d3a93c"}
Sep 18 19:38:34.068884 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.068836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"b8161e005d1270aae16c98d8b2f160158747b9713e315374751b5ae2431c9e51"}
Sep 18 19:38:34.068995 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.068921    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"d1370b8c98481c2082766e9828731cafeb887417fd9013ce766285bc07439656"}
Sep 18 19:38:34.070942 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.070765    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:34.071898 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.071751    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"]
Sep 18 19:38:34.081887 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.080533    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xcshw" podStartSLOduration=2.762355913 podStartE2EDuration="21.080513978s" totalImagesPullingTime="18.318158065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:34.079555253 +0000 UTC m=+22.597524897" watchObservedRunningTime="2026-09-18 19:38:34.080513978 +0000 UTC m=+22.598483614"
Sep 18 19:38:34.091385 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.091344    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" podStartSLOduration=0.09133454 podStartE2EDuration="91.33454ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:34.09129068 +0000 UTC m=+22.609260313" watchObservedRunningTime="2026-09-18 19:38:34.09133454 +0000 UTC m=+22.609304179"
Sep 18 19:38:34.106504 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.106470    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rwjqg" podStartSLOduration=2.835153013 podStartE2EDuration="21.106458193s" totalImagesPullingTime="18.27130518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:34.106230168 +0000 UTC m=+22.624199801" watchObservedRunningTime="2026-09-18 19:38:34.106458193 +0000 UTC m=+22.624427834"
Sep 18 19:38:34.776916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.776890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:34.777080 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:34.777022    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:34.777080 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.777069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:34.777228 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:34.777178    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:34.910959 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:34.910939    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:35.073106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.072927    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5" event={"ID":"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0","Type":"ContainerStarted","Data":"2c124fcfddbfab069482b1816c28df9d0172e36dcc7ed194f0713994aa36bb22"}
Sep 18 19:38:35.073266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.073140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5" event={"ID":"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0","Type":"ContainerStarted","Data":"e74c47f0e9c63be667eacc7cb184b24b54706806854c8f6592e347ca20cf59ea"}
Sep 18 19:38:35.075815 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.075787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"487420578a39334ddfb4c7e02045f6b9449ec3ce87b8df542485af95712dbd04"}
Sep 18 19:38:35.075933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.075862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"eda9f52aa8e789a7e0171512c527ce0a635cf7feedf42f764650773c6a5927bc"}
Sep 18 19:38:35.075933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.075915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"8ccd4c8e3826cc49580276b1ed7ba58a644d40e766e0c15e48f2aa88b1863d5a"}
Sep 18 19:38:35.076008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.075966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"f543b0580f980899722fb5a5d228be7ff1adeba799a534c771fac5d82dd1e35e"}
Sep 18 19:38:35.077322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.077298    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fwbh5" event={"ID":"6e010f81-8cd6-4365-83df-7651a560fba1","Type":"ContainerStarted","Data":"62d90e96d88356d8b5fd0306b74b17cf45c44e16c50013579f7e1819d388398a"}
Sep 18 19:38:35.081382 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.081360    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="8a8262b934b9e617172f3da3c169d60e1c2af79651ab0fa86408d38f96a9ae17" exitCode=0
Sep 18 19:38:35.081464 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.081424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"8a8262b934b9e617172f3da3c169d60e1c2af79651ab0fa86408d38f96a9ae17"}
Sep 18 19:38:35.083015 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.082916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wmlkl" event={"ID":"3a37a033-426d-4229-bebf-67a87e8d980a","Type":"ContainerStarted","Data":"dd9487fdff8053148d567c2cadcc54253eb0cfc3999a4f9f03567270d9e8d9ff"}
Sep 18 19:38:35.084653 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.084632    2573 generic.go:356] "Generic (PLEG): container finished" podID="f19f41f7a65d19182efb2f9c1106ad56" containerID="968b62ac73d5f21edaf92fffd1d4e39e95bfef20a0b91205de23507933beffe5" exitCode=0
Sep 18 19:38:35.084735 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.084712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"f19f41f7a65d19182efb2f9c1106ad56","Type":"ContainerDied","Data":"968b62ac73d5f21edaf92fffd1d4e39e95bfef20a0b91205de23507933beffe5"}
Sep 18 19:38:35.086234 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.086191    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cxk6" event={"ID":"0385842a-4021-45f6-9d3d-606a34d90b5e","Type":"ContainerStarted","Data":"26f74a643bbdc87abd2db09d82b558b027791a484f68fc3f8f931d95e49187b7"}
Sep 18 19:38:35.087695 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.087670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-29vx8" event={"ID":"fd7438f0-23d8-42bd-82fc-fd43c3f98c1b","Type":"ContainerStarted","Data":"49f0451bff906646422b5955819aadf37eb4abaf82011915b221bfc7657b7715"}
Sep 18 19:38:35.088127 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.088109    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:35.096901 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.096864    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fwbh5" podStartSLOduration=3.81858639 podStartE2EDuration="22.096852427s" totalImagesPullingTime="18.278266037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.096302622 +0000 UTC m=+23.614272263" watchObservedRunningTime="2026-09-18 19:38:35.096852427 +0000 UTC m=+23.614822066"
Sep 18 19:38:35.099703 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.099683    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:35.099798 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:35.099729    2573 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"
Sep 18 19:38:35.110616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.110578    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7cxk6" podStartSLOduration=3.951744637 podStartE2EDuration="22.110566209s" totalImagesPullingTime="18.158821572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.110263226 +0000 UTC m=+23.628232867" watchObservedRunningTime="2026-09-18 19:38:35.110566209 +0000 UTC m=+23.628535848"
Sep 18 19:38:35.144918 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.144887    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-29vx8" podStartSLOduration=3.936337228 podStartE2EDuration="22.144876061s" totalImagesPullingTime="18.208538833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.144866789 +0000 UTC m=+23.662836427" watchObservedRunningTime="2026-09-18 19:38:35.144876061 +0000 UTC m=+23.662845700"
Sep 18 19:38:35.147355 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.147341    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:35.147774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.147760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:35.183263 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.183227    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wmlkl" podStartSLOduration=3.86173081 podStartE2EDuration="22.183194963s" totalImagesPullingTime="18.321464153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.182773114 +0000 UTC m=+23.700742755" watchObservedRunningTime="2026-09-18 19:38:35.183194963 +0000 UTC m=+23.701164604"
Sep 18 19:38:35.747055 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.746953    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:34.910955125Z","UUID":"5eadded6-435d-43ab-b79f-03645e632312","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:35.749173 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.749144    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:35.749317 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.749187    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:35.777532 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:35.777500    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:35.777672 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:35.777617    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:36.092669 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.092585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"f19f41f7a65d19182efb2f9c1106ad56","Type":"ContainerStarted","Data":"272b32b603eae12ca0e744950a5f7172317cf45d1786fa45e4067e1ec310ed6e"}
Sep 18 19:38:36.095223 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.095164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5" event={"ID":"6dae77fb-13e9-43cb-9e44-4ce5dce5fab0","Type":"ContainerStarted","Data":"c20f6a642fbcfcd0c82929dce371c4fdb3431452ba93321acad550c3d6caf081"}
Sep 18 19:38:36.096094 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.096056    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:36.096412 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.096391    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fwbh5"
Sep 18 19:38:36.112670 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.112627    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" podStartSLOduration=22.112615628 podStartE2EDuration="22.112615628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:36.112487257 +0000 UTC m=+24.630456899" watchObservedRunningTime="2026-09-18 19:38:36.112615628 +0000 UTC m=+24.630585265"
Sep 18 19:38:36.145384 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.145332    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4p7p5" podStartSLOduration=3.261695843 podStartE2EDuration="23.145315734s" totalImagesPullingTime="19.883619891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:36.145228134 +0000 UTC m=+24.663197774" watchObservedRunningTime="2026-09-18 19:38:36.145315734 +0000 UTC m=+24.663285376"
Sep 18 19:38:36.776998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.776921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:36.776998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:36.776954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:36.777182 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:36.777025    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:36.777182 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:36.777166    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:37.101535 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:37.101457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"58233b8a33ab4290487fad62f5c36ed63289fe02ca0a7ea07f0b1c4596340c16"}
Sep 18 19:38:37.777253 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:37.777217    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:37.777400 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:37.777355    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:38.777387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:38.777348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:38.777800 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:38.777476    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:38.777800 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:38.777528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:38.777800 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:38.777638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:39.777676 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:39.777478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:39.778083 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:39.777712    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:40.111655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.111563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" event={"ID":"34becf8e-bd90-4e82-80c0-ff527bee92bd","Type":"ContainerStarted","Data":"939e5b03603e22c6da93248e7d02015feed6855ea48f9bbe3a95cbb2b6c5ba3e"}
Sep 18 19:38:40.111857 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.111809    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:40.113477 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.113455    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="f2dda39e6add1d16f67c59adbbaa8a25f5d83f4b78f1af28addf1ef29daaaac6" exitCode=0
Sep 18 19:38:40.113607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.113486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"f2dda39e6add1d16f67c59adbbaa8a25f5d83f4b78f1af28addf1ef29daaaac6"}
Sep 18 19:38:40.127492 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.127473    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:40.136684 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.136648    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn" podStartSLOduration=8.299046798 podStartE2EDuration="27.136636382s" totalImagesPullingTime="18.837589584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:40.136177985 +0000 UTC m=+28.654147628" watchObservedRunningTime="2026-09-18 19:38:40.136636382 +0000 UTC m=+28.654606023"
Sep 18 19:38:40.777672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.777646    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:40.777789 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:40.777743    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:40.778081 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:40.777794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:40.778081 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:40.777890    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:41.117218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.117119    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="cd98fbe84ce500d47e5efd7a7384aa2ab88607b35bade67694f7bac7fe4b3dbe" exitCode=0
Sep 18 19:38:41.117377 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.117214    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"cd98fbe84ce500d47e5efd7a7384aa2ab88607b35bade67694f7bac7fe4b3dbe"}
Sep 18 19:38:41.117611 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.117590    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:41.117710 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.117658    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:41.132302 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.132285    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:38:41.777846 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:41.777816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:41.778190 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:41.777909    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:42.121778 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:42.121696    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="b0392a436b00e6579959202849ac4d5456ff600474621c11155420b23b38cd94" exitCode=0
Sep 18 19:38:42.121908 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:42.121785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"b0392a436b00e6579959202849ac4d5456ff600474621c11155420b23b38cd94"}
Sep 18 19:38:42.777606 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:42.777568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:42.777606 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:42.777605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:42.777815 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:42.777680    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:42.777877 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:42.777852    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:43.777874 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:43.777843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:43.778465 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:43.777944    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:44.777634 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:44.777604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:44.777634 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:44.777632    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:44.777845 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:44.777697    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:44.777887 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:44.777841    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:45.777285 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:45.777247    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:45.777713 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:45.777392    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:46.427931 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:46.427899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:46.428115 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:46.427941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:46.428115 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.428031    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:46.428115 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.428055    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:46.428115 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.428077    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret podName:a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.428064717 +0000 UTC m=+66.946034336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret") pod "global-pull-secret-syncer-f75sh" (UID: "a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:46.428115 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.428117    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs podName:87591871-d05d-4e65-ba24-915d085bf6b2 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.428100724 +0000 UTC m=+66.946070350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs") pod "network-metrics-daemon-8grvn" (UID: "87591871-d05d-4e65-ba24-915d085bf6b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:46.528883 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:46.528851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:46.529028 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.529009    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:46.529066 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.529034    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:46.529066 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.529045    2573 projected.go:196] Error preparing data for projected volume kube-api-access-g6rzp for pod openshift-network-diagnostics/network-check-target-kh7cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:46.529126 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.529094    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp podName:52504afc-712e-48f5-a8d4-586460e326c5 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:18.529081396 +0000 UTC m=+67.047051014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g6rzp" (UniqueName: "kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp") pod "network-check-target-kh7cd" (UID: "52504afc-712e-48f5-a8d4-586460e326c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:46.777948 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:46.777919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:46.778422 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.778019    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:46.778422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:46.778045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:46.778422 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:46.778091    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:47.135761 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:47.135729    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="d293910b33d6edad253613404bf99d3ea1ff194a9a507b32ab3557e4ab5dcd13" exitCode=0
Sep 18 19:38:47.135945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:47.135773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"d293910b33d6edad253613404bf99d3ea1ff194a9a507b32ab3557e4ab5dcd13"}
Sep 18 19:38:47.780604 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:47.780574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:47.780976 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:47.780668    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:48.141330 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:48.141297    2573 generic.go:356] "Generic (PLEG): container finished" podID="4e1f3ce1-13e9-4c68-afca-dc5e17918cac" containerID="909338bb355255fae971331240ea31ee695c2bca4615ffb7243c21667708049a" exitCode=0
Sep 18 19:38:48.141516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:48.141344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerDied","Data":"909338bb355255fae971331240ea31ee695c2bca4615ffb7243c21667708049a"}
Sep 18 19:38:48.777038 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:48.777008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:48.777253 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:48.777015    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:48.777253 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:48.777100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:48.777341 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:48.777246    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:49.147129 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:49.147096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" event={"ID":"4e1f3ce1-13e9-4c68-afca-dc5e17918cac","Type":"ContainerStarted","Data":"660a244a0ec17be1e87e65a8a8a50eb8f439bca763fd97ae0f152947da1218ff"}
Sep 18 19:38:49.169195 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:49.169144    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kx2s7" podStartSLOduration=8.196613871 podStartE2EDuration="36.169129947s" totalImagesPullingTime="27.972516076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:49.16794806 +0000 UTC m=+37.685917700" watchObservedRunningTime="2026-09-18 19:38:49.169129947 +0000 UTC m=+37.687099587"
Sep 18 19:38:49.777802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:49.777763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:49.777980 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:49.777897    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:50.777496 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:50.777465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:50.777909 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:50.777465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:50.777909 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:50.777566    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:50.777909 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:50.777666    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:51.776951 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:51.776919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:51.777178 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:51.777051    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:52.777424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:52.777380    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:52.777874 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:52.777515    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:52.777874 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:52.777563    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:52.777874 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:52.777653    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:53.778229 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:53.778186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:53.778594 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:53.778283    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:54.278984 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.278791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f75sh"]
Sep 18 19:38:54.279136 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.279063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:54.279218 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:54.279174    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:54.281727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.281689    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kh7cd"]
Sep 18 19:38:54.281850 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.281781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:54.281911 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:54.281861    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:54.282327 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.282301    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8grvn"]
Sep 18 19:38:54.282803 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:54.282407    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:54.282803 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:54.282500    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:55.777590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:55.777561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:55.777975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:55.777561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:55.777975 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:55.777657    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:55.777975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:55.777682    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:55.777975 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:55.777760    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:55.777975 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:55.777812    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:57.777922 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:57.777892    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:57.777922 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:57.777918    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:57.778571 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:57.777981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f75sh" podUID="a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d"
Sep 18 19:38:57.778571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:57.778075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:57.778571 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:57.778177    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kh7cd" podUID="52504afc-712e-48f5-a8d4-586460e326c5"
Sep 18 19:38:57.778571 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:38:57.778345    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8grvn" podUID="87591871-d05d-4e65-ba24-915d085bf6b2"
Sep 18 19:38:58.369964 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.369931    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeReady"
Sep 18 19:38:58.370142 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.370074    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:38:58.404689 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.404661    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:38:58.431286 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.431215    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wgnm9"]
Sep 18 19:38:58.431431 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.431300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.433929 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.433902    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:38:58.434076 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.433949    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nzlpc\""
Sep 18 19:38:58.434076 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.434009    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:38:58.434238 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.434073    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:38:58.439172 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.439152    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:38:58.449130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.449110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:38:58.449237 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.449140    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mzbnp"]
Sep 18 19:38:58.449282 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.449242    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.452233 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.451754    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:38:58.452712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.452362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hn8v8\""
Sep 18 19:38:58.452712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.452431    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:38:58.452712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.452505    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:38:58.471346 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.471329    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wgnm9"]
Sep 18 19:38:58.471442 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.471355    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzbnp"]
Sep 18 19:38:58.471488 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.471440    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.473814 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.473792    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:38:58.473905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.473817    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:38:58.473905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.473853    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j2prk\""
Sep 18 19:38:58.621138 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-cert\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.621269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l85t\" (UniqueName: \"kubernetes.io/projected/7a2b66dd-9172-492c-94cd-0f61949f8759-kube-api-access-9l85t\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.621269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621433 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b66dd-9172-492c-94cd-0f61949f8759-config-volume\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.621433 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621433 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2b66dd-9172-492c-94cd-0f61949f8759-tmp-dir\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.621433 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4nh\" (UniqueName: \"kubernetes.io/projected/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-kube-api-access-9m4nh\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn7s\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a2b66dd-9172-492c-94cd-0f61949f8759-metrics-tls\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.621672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.621601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722576 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722576 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4nh\" (UniqueName: \"kubernetes.io/projected/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-kube-api-access-9m4nh\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.722730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn7s\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a2b66dd-9172-492c-94cd-0f61949f8759-metrics-tls\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.722730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.722923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.722830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-cert\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.723058 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723016    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l85t\" (UniqueName: \"kubernetes.io/projected/7a2b66dd-9172-492c-94cd-0f61949f8759-kube-api-access-9l85t\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.723058 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.723152 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b66dd-9172-492c-94cd-0f61949f8759-config-volume\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.723152 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723127    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.723289 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2b66dd-9172-492c-94cd-0f61949f8759-tmp-dir\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.723520 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723503    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a2b66dd-9172-492c-94cd-0f61949f8759-tmp-dir\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.723600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.724020 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.723997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2b66dd-9172-492c-94cd-0f61949f8759-config-volume\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.724224 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.724186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.724579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.724558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.726875 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.726855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.727040 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.727019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.727360 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.727339    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a2b66dd-9172-492c-94cd-0f61949f8759-metrics-tls\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.727464 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.727402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-cert\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.727524 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.727462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.730028 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.730008    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn7s\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.730169 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.730150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l85t\" (UniqueName: \"kubernetes.io/projected/7a2b66dd-9172-492c-94cd-0f61949f8759-kube-api-access-9l85t\") pod \"dns-default-mzbnp\" (UID: \"7a2b66dd-9172-492c-94cd-0f61949f8759\") " pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:58.730330 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.730311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4nh\" (UniqueName: \"kubernetes.io/projected/233b6f4b-32f4-4bf3-ad0b-878dfd95d556-kube-api-access-9m4nh\") pod \"ingress-canary-wgnm9\" (UID: \"233b6f4b-32f4-4bf3-ad0b-878dfd95d556\") " pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.731003 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.730989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token\") pod \"image-registry-9698db596-t5mrz\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") " pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.741079 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.741062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:38:58.758858 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.758842    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wgnm9"
Sep 18 19:38:58.779144 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:58.779010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:38:59.070590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.070506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wgnm9"]
Sep 18 19:38:59.074686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.074659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:38:59.074993 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:59.074913    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233b6f4b_32f4_4bf3_ad0b_878dfd95d556.slice/crio-8bb01878ea2cbf6af00347be947d59c33eb15f554f5ef54e728efa9abfbebd0f WatchSource:0}: Error finding container 8bb01878ea2cbf6af00347be947d59c33eb15f554f5ef54e728efa9abfbebd0f: Status 404 returned error can't find the container with id 8bb01878ea2cbf6af00347be947d59c33eb15f554f5ef54e728efa9abfbebd0f
Sep 18 19:38:59.078295 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:59.078274    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbc201f_a801_49fc_a806_66bcd971853d.slice/crio-b512f295f7ee00e09b087f3087143fc868bec73cf27b204bb01a55847811daaf WatchSource:0}: Error finding container b512f295f7ee00e09b087f3087143fc868bec73cf27b204bb01a55847811daaf: Status 404 returned error can't find the container with id b512f295f7ee00e09b087f3087143fc868bec73cf27b204bb01a55847811daaf
Sep 18 19:38:59.095107 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.095088    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mzbnp"]
Sep 18 19:38:59.097652 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:38:59.097631    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2b66dd_9172_492c_94cd_0f61949f8759.slice/crio-6aef9805adcf52f38bfb9968aa2be4672782a0a4181ba2dda19c7ce8671a2e14 WatchSource:0}: Error finding container 6aef9805adcf52f38bfb9968aa2be4672782a0a4181ba2dda19c7ce8671a2e14: Status 404 returned error can't find the container with id 6aef9805adcf52f38bfb9968aa2be4672782a0a4181ba2dda19c7ce8671a2e14
Sep 18 19:38:59.170284 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.170262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzbnp" event={"ID":"7a2b66dd-9172-492c-94cd-0f61949f8759","Type":"ContainerStarted","Data":"6aef9805adcf52f38bfb9968aa2be4672782a0a4181ba2dda19c7ce8671a2e14"}
Sep 18 19:38:59.171408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.171386    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wgnm9" event={"ID":"233b6f4b-32f4-4bf3-ad0b-878dfd95d556","Type":"ContainerStarted","Data":"8bb01878ea2cbf6af00347be947d59c33eb15f554f5ef54e728efa9abfbebd0f"}
Sep 18 19:38:59.172507 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.172487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9698db596-t5mrz" event={"ID":"0dbc201f-a801-49fc-a806-66bcd971853d","Type":"ContainerStarted","Data":"b512f295f7ee00e09b087f3087143fc868bec73cf27b204bb01a55847811daaf"}
Sep 18 19:38:59.777019 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.776905    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:38:59.777837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.776906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:38:59.777837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.776908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:38:59.781362 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781339    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:38:59.782056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781371    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vm6\""
Sep 18 19:38:59.782056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781401    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:38:59.782056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781552    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:38:59.782056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781622    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:38:59.782056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:38:59.781796    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9245\""
Sep 18 19:39:00.177301 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:00.177268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9698db596-t5mrz" event={"ID":"0dbc201f-a801-49fc-a806-66bcd971853d","Type":"ContainerStarted","Data":"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"}
Sep 18 19:39:00.177484 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:00.177341    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:39:00.196067 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:00.196013    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9698db596-t5mrz" podStartSLOduration=69.195945219 podStartE2EDuration="1m9.195945219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:00.195385366 +0000 UTC m=+48.713355009" watchObservedRunningTime="2026-09-18 19:39:00.195945219 +0000 UTC m=+48.713914859"
Sep 18 19:39:02.185322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.185079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzbnp" event={"ID":"7a2b66dd-9172-492c-94cd-0f61949f8759","Type":"ContainerStarted","Data":"5dd2d0719f0c2f794451828d85f2462744df3cc6690595febb745936aa0ed7ce"}
Sep 18 19:39:02.185810 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.185327    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:39:02.185810 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.185356    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mzbnp" event={"ID":"7a2b66dd-9172-492c-94cd-0f61949f8759","Type":"ContainerStarted","Data":"b484363726678fd0d5a852e697c96f2791c11b0d6d0719e87b7cdaea16fb658d"}
Sep 18 19:39:02.186689 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.186663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wgnm9" event={"ID":"233b6f4b-32f4-4bf3-ad0b-878dfd95d556","Type":"ContainerStarted","Data":"7fc2ceb598bf6014f93828ec12326f8c77dde3801e3d906573ba874838e1f1f9"}
Sep 18 19:39:02.203751 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.203706    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mzbnp" podStartSLOduration=1.422843315 podStartE2EDuration="4.203691536s" totalImagesPullingTime="2.780848221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:02.202850272 +0000 UTC m=+50.720819949" watchObservedRunningTime="2026-09-18 19:39:02.203691536 +0000 UTC m=+50.721661182"
Sep 18 19:39:02.217899 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:02.217857    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wgnm9" podStartSLOduration=1.4053797829999999 podStartE2EDuration="4.217842942s" totalImagesPullingTime="2.812463159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:02.217461477 +0000 UTC m=+50.735431121" watchObservedRunningTime="2026-09-18 19:39:02.217842942 +0000 UTC m=+50.735812582"
Sep 18 19:39:07.565926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.565890    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-64wpr"]
Sep 18 19:39:07.568736 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.568714    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.570977 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.570949    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:39:07.571212 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.571186    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wglrh\""
Sep 18 19:39:07.572030 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.572009    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:39:07.572108 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.572035    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:39:07.572108 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.572067    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:39:07.581751 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.581733    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-64wpr"]
Sep 18 19:39:07.589278 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.589257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.589361 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.589318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtcv\" (UniqueName: \"kubernetes.io/projected/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-api-access-zdtcv\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.589361 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.589349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1904fe7-40ac-46fe-a615-39ce797ef1c1-data-volume\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.589432 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.589385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.589462 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.589438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1904fe7-40ac-46fe-a615-39ce797ef1c1-crio-socket\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.690267 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.690242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtcv\" (UniqueName: \"kubernetes.io/projected/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-api-access-zdtcv\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.690598 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.690281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1904fe7-40ac-46fe-a615-39ce797ef1c1-data-volume\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.690824 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.690805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.690954 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.690939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1904fe7-40ac-46fe-a615-39ce797ef1c1-crio-socket\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.691054 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.691038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.691184 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.691159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1904fe7-40ac-46fe-a615-39ce797ef1c1-crio-socket\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.691336 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:07.690994    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:07.691507 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:07.691496    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.191467867 +0000 UTC m=+56.709437499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:07.691783 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.691068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1904fe7-40ac-46fe-a615-39ce797ef1c1-data-volume\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:07.691891 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:07.691159    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:07.692008 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:07.692000    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:08.191983137 +0000 UTC m=+56.709952773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:07.700228 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:07.700196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtcv\" (UniqueName: \"kubernetes.io/projected/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-api-access-zdtcv\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:08.193998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:08.193959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:08.194149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:08.194054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:08.194149 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:08.194108    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:08.194149 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:08.194133    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:08.194277 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:08.194169    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.194155238 +0000 UTC m=+57.712124857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:08.194277 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:08.194186    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.194174844 +0000 UTC m=+57.712144479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:08.957637 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:08.957615    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzbnp_7a2b66dd-9172-492c-94cd-0f61949f8759/dns/0.log"
Sep 18 19:39:09.137482 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:09.137457    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzbnp_7a2b66dd-9172-492c-94cd-0f61949f8759/kube-rbac-proxy/0.log"
Sep 18 19:39:09.200772 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:09.200745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:09.200913 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:09.200782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:09.200913 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:09.200865    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.200913 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:09.200892    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.201022 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:09.200928    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:11.200915164 +0000 UTC m=+59.718884782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.201022 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:09.200955    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls podName:b1904fe7-40ac-46fe-a615-39ce797ef1c1 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:11.200945282 +0000 UTC m=+59.718914900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64wpr" (UID: "b1904fe7-40ac-46fe-a615-39ce797ef1c1") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.336166 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:09.336107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29vx8_fd7438f0-23d8-42bd-82fc-fd43c3f98c1b/dns-node-resolver/0.log"
Sep 18 19:39:10.136214 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:10.136175    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9698db596-t5mrz_0dbc201f-a801-49fc-a806-66bcd971853d/registry/0.log"
Sep 18 19:39:10.335884 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:10.335854    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cxk6_0385842a-4021-45f6-9d3d-606a34d90b5e/node-ca/0.log"
Sep 18 19:39:11.213316 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.213284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:11.213671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.213323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:11.213815 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.213798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1904fe7-40ac-46fe-a615-39ce797ef1c1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:11.215444 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.215422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1904fe7-40ac-46fe-a615-39ce797ef1c1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64wpr\" (UID: \"b1904fe7-40ac-46fe-a615-39ce797ef1c1\") " pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:11.477612 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.477542    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-64wpr"
Sep 18 19:39:11.536776 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.536753    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wgnm9_233b6f4b-32f4-4bf3-ad0b-878dfd95d556/serve-healthcheck-canary/0.log"
Sep 18 19:39:11.781670 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:11.781608    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-64wpr"]
Sep 18 19:39:11.785188 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:11.785156    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1904fe7_40ac_46fe_a615_39ce797ef1c1.slice/crio-fe9f403756a75daa8861f6b8d2b525e8ed9762edbdd5a08eee38983d4261b379 WatchSource:0}: Error finding container fe9f403756a75daa8861f6b8d2b525e8ed9762edbdd5a08eee38983d4261b379: Status 404 returned error can't find the container with id fe9f403756a75daa8861f6b8d2b525e8ed9762edbdd5a08eee38983d4261b379
Sep 18 19:39:12.191303 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:12.191274    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mzbnp"
Sep 18 19:39:12.211155 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:12.211131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64wpr" event={"ID":"b1904fe7-40ac-46fe-a615-39ce797ef1c1","Type":"ContainerStarted","Data":"16281e52432a836522a216c7cb40f5da80c91da9408606485d3db0de3bc57e6c"}
Sep 18 19:39:12.211288 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:12.211178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64wpr" event={"ID":"b1904fe7-40ac-46fe-a615-39ce797ef1c1","Type":"ContainerStarted","Data":"fe9f403756a75daa8861f6b8d2b525e8ed9762edbdd5a08eee38983d4261b379"}
Sep 18 19:39:13.136318 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:13.136289    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ds2kn"
Sep 18 19:39:13.217382 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:13.217344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64wpr" event={"ID":"b1904fe7-40ac-46fe-a615-39ce797ef1c1","Type":"ContainerStarted","Data":"4a6a4a71d36ffb077e754e1b05384026934c78708fb4785a4f8da5750a6c4d55"}
Sep 18 19:39:14.222657 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:14.222617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64wpr" event={"ID":"b1904fe7-40ac-46fe-a615-39ce797ef1c1","Type":"ContainerStarted","Data":"12bbfe563c5061f2b265b2c73433c43863246c859ffd76958a758b12901a99c9"}
Sep 18 19:39:14.238516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:14.238475    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-64wpr" podStartSLOduration=5.356570138 podStartE2EDuration="7.238461675s" totalImagesPullingTime="1.881891537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:14.237995189 +0000 UTC m=+62.755964830" watchObservedRunningTime="2026-09-18 19:39:14.238461675 +0000 UTC m=+62.756431314"
Sep 18 19:39:15.271368 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.271340    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hlhrv"]
Sep 18 19:39:15.273437 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.273421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.275544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275522    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:39:15.275544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275522    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:39:15.275711 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275657    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:39:15.275908 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275891    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:39:15.275978 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275934    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:39:15.275978 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275953    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2j4z\""
Sep 18 19:39:15.275978 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.275972    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:39:15.285904 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.285883    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-swcc4"]
Sep 18 19:39:15.288063 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.288049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.292800 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.292777    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 19:39:15.292902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.292874    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 19:39:15.293317 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.293119    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 19:39:15.293421 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.293404    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wkskz\""
Sep 18 19:39:15.320584 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.320565    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-swcc4"]
Sep 18 19:39:15.341242 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqtj\" (UniqueName: \"kubernetes.io/projected/7962429b-25d2-4e9f-ad55-e8a01ab50822-kube-api-access-rpqtj\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-root\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.341353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.341509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.341509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341377    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.341509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-metrics-client-ca\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54cn\" (UniqueName: \"kubernetes.io/projected/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-api-access-q54cn\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.341721 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-sys\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341721 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-wtmp\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341721 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341696    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-textfile\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.341837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.341754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.442540 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-root\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-metrics-client-ca\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:15.442686    2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 18 19:39:15.442743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442621    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-root\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:15.442755    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls podName:95c56be6-2ba8-494c-b2a6-610c3e25fe4e nodeName:}" failed. No retries permitted until 2026-09-18 19:39:15.942735434 +0000 UTC m=+64.460705076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-swcc4" (UID: "95c56be6-2ba8-494c-b2a6-610c3e25fe4e") : secret "kube-state-metrics-tls" not found
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54cn\" (UniqueName: \"kubernetes.io/projected/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-api-access-q54cn\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-sys\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-wtmp\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-textfile\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442936    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.442960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqtj\" (UniqueName: \"kubernetes.io/projected/7962429b-25d2-4e9f-ad55-e8a01ab50822-kube-api-access-rpqtj\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-sys\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-wtmp\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:15.443323    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443340    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-metrics-client-ca\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:15.443382    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls podName:7962429b-25d2-4e9f-ad55-e8a01ab50822 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:15.94336562 +0000 UTC m=+64.461335255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls") pod "node-exporter-hlhrv" (UID: "7962429b-25d2-4e9f-ad55-e8a01ab50822") : secret "node-exporter-tls" not found
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.443671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.443883 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-textfile\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.443883 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.443739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.445144 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.445122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.445251 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.445228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.453017 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.452998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqtj\" (UniqueName: \"kubernetes.io/projected/7962429b-25d2-4e9f-ad55-e8a01ab50822-kube-api-access-rpqtj\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.453213 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.453185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54cn\" (UniqueName: \"kubernetes.io/projected/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-api-access-q54cn\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.945992 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.945935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.946189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.946040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:15.948307 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.948277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7962429b-25d2-4e9f-ad55-e8a01ab50822-node-exporter-tls\") pod \"node-exporter-hlhrv\" (UID: \"7962429b-25d2-4e9f-ad55-e8a01ab50822\") " pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:15.948516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:15.948493    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c56be6-2ba8-494c-b2a6-610c3e25fe4e-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-swcc4\" (UID: \"95c56be6-2ba8-494c-b2a6-610c3e25fe4e\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:16.122096 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.122065    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:39:16.126443 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.126405    2573 patch_prober.go:28] interesting pod/image-registry-9698db596-t5mrz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 19:39:16.126559 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.126465    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-9698db596-t5mrz" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:39:16.181950 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.181924    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlhrv"
Sep 18 19:39:16.190957 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:16.190931    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7962429b_25d2_4e9f_ad55_e8a01ab50822.slice/crio-2b17c0d66408df108ae930005068607024ad179d540415a2d347f6e5ac41c5ed WatchSource:0}: Error finding container 2b17c0d66408df108ae930005068607024ad179d540415a2d347f6e5ac41c5ed: Status 404 returned error can't find the container with id 2b17c0d66408df108ae930005068607024ad179d540415a2d347f6e5ac41c5ed
Sep 18 19:39:16.196129 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.196089    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4"
Sep 18 19:39:16.230961 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.230932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlhrv" event={"ID":"7962429b-25d2-4e9f-ad55-e8a01ab50822","Type":"ContainerStarted","Data":"2b17c0d66408df108ae930005068607024ad179d540415a2d347f6e5ac41c5ed"}
Sep 18 19:39:16.367661 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.367635    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:16.370482 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.370465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.373301 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373272    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:39:16.373534 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373507    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:39:16.373664 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373509    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:39:16.373664 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373581    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:39:16.373664 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:39:16.373664 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:39:16.373963 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373887    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:39:16.373963 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373910    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:39:16.373963 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373949    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:39:16.373963 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.373952    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rkpdl\""
Sep 18 19:39:16.385397 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.385377    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:16.449997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.449937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.449997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.449971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450147 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450147 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450049    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450147 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450147 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450147 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450417 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450417 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450495 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450537 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450537 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdql\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.450622 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.450557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.462289 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.462267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-swcc4"]
Sep 18 19:39:16.465073 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:16.465053    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c56be6_2ba8_494c_b2a6_610c3e25fe4e.slice/crio-92ea18b8ed4ce5340d2e50d41c2bee518297c2e200fa3094b6170a4029cff4d2 WatchSource:0}: Error finding container 92ea18b8ed4ce5340d2e50d41c2bee518297c2e200fa3094b6170a4029cff4d2: Status 404 returned error can't find the container with id 92ea18b8ed4ce5340d2e50d41c2bee518297c2e200fa3094b6170a4029cff4d2
Sep 18 19:39:16.551083 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551186 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551186 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551320 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551320 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdql\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551320 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551462 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551515 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551479    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551674 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:16.551655    2573 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 19:39:16.551753 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:16.551744    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls podName:d46b3a1b-e45c-4823-8c53-db423ffa9570 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.051715236 +0000 UTC m=+65.569684855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570") : secret "alertmanager-main-tls" not found
Sep 18 19:39:16.551938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.551938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.551984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.552024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.552047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.552072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.552096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.552380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.552350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.554174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.554148    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.554291 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.554177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.554582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.554561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.554763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.554742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.555014 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.554991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.555086 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.555056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.555086 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.555073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.555236 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.555220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:16.558672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:16.558654    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdql\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:17.056673 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.056647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:17.058823 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.058805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:17.237061 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.237022    2573 generic.go:356] "Generic (PLEG): container finished" podID="7962429b-25d2-4e9f-ad55-e8a01ab50822" containerID="cb03063d5648ef07fd6e0f33d57a8e0a7194e141107e5bdb2a3c49856120067e" exitCode=0
Sep 18 19:39:17.237239 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.237116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlhrv" event={"ID":"7962429b-25d2-4e9f-ad55-e8a01ab50822","Type":"ContainerDied","Data":"cb03063d5648ef07fd6e0f33d57a8e0a7194e141107e5bdb2a3c49856120067e"}
Sep 18 19:39:17.238577 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.238557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4" event={"ID":"95c56be6-2ba8-494c-b2a6-610c3e25fe4e","Type":"ContainerStarted","Data":"92ea18b8ed4ce5340d2e50d41c2bee518297c2e200fa3094b6170a4029cff4d2"}
Sep 18 19:39:17.279935 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.279907    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:39:17.281569 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.281545    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"]
Sep 18 19:39:17.306314 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.306282    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"]
Sep 18 19:39:17.306420 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.306404    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.309043 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.308964    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-51rqr2jujkche\""
Sep 18 19:39:17.309155 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.309134    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:39:17.310930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.309677    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lftqq\""
Sep 18 19:39:17.310930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.309880    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:39:17.310930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.310054    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:39:17.310930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.310315    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:39:17.310930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.310522    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:39:17.358976 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.358947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359042 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.358994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359042 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmmh\" (UniqueName: \"kubernetes.io/projected/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-kube-api-access-fsmmh\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359178 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-grpc-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359178 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-metrics-client-ca\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359284 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359229    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.359284 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.359258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460387 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460715    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmmh\" (UniqueName: \"kubernetes.io/projected/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-kube-api-access-fsmmh\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-grpc-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.460938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.460849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-metrics-client-ca\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.461587 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.461542    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-metrics-client-ca\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.463705 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.463679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.463852 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.463824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.464542 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.464313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.464542 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.464434    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.464697 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.464626    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.465349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.465317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-secret-grpc-tls\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.469038 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.469016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmmh\" (UniqueName: \"kubernetes.io/projected/e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0-kube-api-access-fsmmh\") pod \"thanos-querier-55c6b56b78-5q7rx\" (UID: \"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0\") " pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:17.582397 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.582317    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:39:17.585544 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:17.585517    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46b3a1b_e45c_4823_8c53_db423ffa9570.slice/crio-51077128615ae4503d1c317fac74dd75084b379758f175c25ecd3ada7d25b612 WatchSource:0}: Error finding container 51077128615ae4503d1c317fac74dd75084b379758f175c25ecd3ada7d25b612: Status 404 returned error can't find the container with id 51077128615ae4503d1c317fac74dd75084b379758f175c25ecd3ada7d25b612
Sep 18 19:39:17.618310 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:17.618290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:18.242945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.242911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"51077128615ae4503d1c317fac74dd75084b379758f175c25ecd3ada7d25b612"}
Sep 18 19:39:18.244746 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.244724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlhrv" event={"ID":"7962429b-25d2-4e9f-ad55-e8a01ab50822","Type":"ContainerStarted","Data":"636f06c8c492807a4b6701b19a97678424ced6ac92c3c743d28fc83ef07788ab"}
Sep 18 19:39:18.244855 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.244762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlhrv" event={"ID":"7962429b-25d2-4e9f-ad55-e8a01ab50822","Type":"ContainerStarted","Data":"dfb8aaf12a0771422d9316fbe5cdae395a5f1b919efeae89dae7675c3168c98e"}
Sep 18 19:39:18.264876 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.264791    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hlhrv" podStartSLOduration=2.362012803 podStartE2EDuration="3.264735588s" totalImagesPullingTime="844.454046ms" totalInitContainerRuntime="58.268739ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:18.264242885 +0000 UTC m=+66.782212524" watchObservedRunningTime="2026-09-18 19:39:18.264735588 +0000 UTC m=+66.782705227"
Sep 18 19:39:18.282133 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.282111    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"]
Sep 18 19:39:18.327752 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:18.327721    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5baf44d_2bd1_4deb_9c4e_91a7d71a6ea0.slice/crio-2f7ee924404fbc52f042dee70700d7be496aa97232c3b29e784cbfddaba4737c WatchSource:0}: Error finding container 2f7ee924404fbc52f042dee70700d7be496aa97232c3b29e784cbfddaba4737c: Status 404 returned error can't find the container with id 2f7ee924404fbc52f042dee70700d7be496aa97232c3b29e784cbfddaba4737c
Sep 18 19:39:18.468900 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.468866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:39:18.469372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.468984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:39:18.471534 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.471506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:18.471702 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.471506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:18.481782 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.481711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87591871-d05d-4e65-ba24-915d085bf6b2-metrics-certs\") pod \"network-metrics-daemon-8grvn\" (UID: \"87591871-d05d-4e65-ba24-915d085bf6b2\") " pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:39:18.482826 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.482798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d-original-pull-secret\") pod \"global-pull-secret-syncer-f75sh\" (UID: \"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d\") " pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:39:18.570096 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.570067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:39:18.573109 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.572880    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:18.583008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.582827    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:18.599218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.597430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rzp\" (UniqueName: \"kubernetes.io/projected/52504afc-712e-48f5-a8d4-586460e326c5-kube-api-access-g6rzp\") pod \"network-check-target-kh7cd\" (UID: \"52504afc-712e-48f5-a8d4-586460e326c5\") " pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:39:18.692761 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.692732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f75sh"
Sep 18 19:39:18.701052 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.701009    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9245\""
Sep 18 19:39:18.707773 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.707754    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vm6\""
Sep 18 19:39:18.709441 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.709413    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8grvn"
Sep 18 19:39:18.717034 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:18.716993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:39:19.232653 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.232344    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kh7cd"]
Sep 18 19:39:19.237768 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:19.237742    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52504afc_712e_48f5_a8d4_586460e326c5.slice/crio-8762b31a1867cdf0e5784c5bd6d5fbe908bfc5ae62765bc8e824973b8b9b9786 WatchSource:0}: Error finding container 8762b31a1867cdf0e5784c5bd6d5fbe908bfc5ae62765bc8e824973b8b9b9786: Status 404 returned error can't find the container with id 8762b31a1867cdf0e5784c5bd6d5fbe908bfc5ae62765bc8e824973b8b9b9786
Sep 18 19:39:19.246003 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.245932    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8grvn"]
Sep 18 19:39:19.247582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.247170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f75sh"]
Sep 18 19:39:19.252972 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.252908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kh7cd" event={"ID":"52504afc-712e-48f5-a8d4-586460e326c5","Type":"ContainerStarted","Data":"8762b31a1867cdf0e5784c5bd6d5fbe908bfc5ae62765bc8e824973b8b9b9786"}
Sep 18 19:39:19.255481 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:19.254899    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87591871_d05d_4e65_ba24_915d085bf6b2.slice/crio-b6ea16d2a46a734901bdeca6b655dc48ed087a35318a5417e120710ecdc8980f WatchSource:0}: Error finding container b6ea16d2a46a734901bdeca6b655dc48ed087a35318a5417e120710ecdc8980f: Status 404 returned error can't find the container with id b6ea16d2a46a734901bdeca6b655dc48ed087a35318a5417e120710ecdc8980f
Sep 18 19:39:19.256961 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.256912    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac" exitCode=0
Sep 18 19:39:19.257030 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.256975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"}
Sep 18 19:39:19.257454 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:19.257417    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8290d9b_8eba_4ca6_9452_1a9fcf6e8a4d.slice/crio-34c03be1bf81e48a39fa5cfdd617559b72880857c001da1e6ee9803e71a06a9d WatchSource:0}: Error finding container 34c03be1bf81e48a39fa5cfdd617559b72880857c001da1e6ee9803e71a06a9d: Status 404 returned error can't find the container with id 34c03be1bf81e48a39fa5cfdd617559b72880857c001da1e6ee9803e71a06a9d
Sep 18 19:39:19.267242 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.267215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"2f7ee924404fbc52f042dee70700d7be496aa97232c3b29e784cbfddaba4737c"}
Sep 18 19:39:19.270492 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.270466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4" event={"ID":"95c56be6-2ba8-494c-b2a6-610c3e25fe4e","Type":"ContainerStarted","Data":"979806de12dfaf7e36b40aa33bbb38ba7e45a6d88bbd1f7b610c0154bc535606"}
Sep 18 19:39:19.270587 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.270520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4" event={"ID":"95c56be6-2ba8-494c-b2a6-610c3e25fe4e","Type":"ContainerStarted","Data":"9fcb2fca6ee4fb0552db186f0294a87ad6b55f1e8d02e3940801ac219e319d9a"}
Sep 18 19:39:19.270587 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.270548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4" event={"ID":"95c56be6-2ba8-494c-b2a6-610c3e25fe4e","Type":"ContainerStarted","Data":"34788e956cb7d53cba78e6ad180abad21bc18c51287786bd290f96055c7e9503"}
Sep 18 19:39:19.309732 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:19.309682    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-swcc4" podStartSLOduration=2.423768404 podStartE2EDuration="4.309663099s" totalImagesPullingTime="1.885894695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:19.308557437 +0000 UTC m=+67.826527091" watchObservedRunningTime="2026-09-18 19:39:19.309663099 +0000 UTC m=+67.827632741"
Sep 18 19:39:20.277894 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:20.277799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8grvn" event={"ID":"87591871-d05d-4e65-ba24-915d085bf6b2","Type":"ContainerStarted","Data":"b6ea16d2a46a734901bdeca6b655dc48ed087a35318a5417e120710ecdc8980f"}
Sep 18 19:39:20.280741 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:20.280698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f75sh" event={"ID":"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d","Type":"ContainerStarted","Data":"34c03be1bf81e48a39fa5cfdd617559b72880857c001da1e6ee9803e71a06a9d"}
Sep 18 19:39:21.523010 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.521008    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:21.532595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.529820    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.532595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.531966    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:39:21.532595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.532227    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:39:21.532897 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.532795    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:39:21.532897 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.532795    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-63drttkr3886n\""
Sep 18 19:39:21.533195 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533173    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9sfxp\""
Sep 18 19:39:21.533330 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533236    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:39:21.533403 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533361    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:39:21.533459 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533603    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533785    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533175    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.533977    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.534093    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.534360    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:39:21.534500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.534398    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:39:21.536723 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.536388    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:39:21.538999 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.538981    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:21.541442 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.541273    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.596678 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.596972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s22d\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.597000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.597025    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.597062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597734 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.597144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.597734 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.597175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.697847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.697996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.698445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.698192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700293 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700023    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700499 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s22d\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700658    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700838 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.700838 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.700724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.701116 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.701096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.701626 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.701601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.705747 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.703092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.705747 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.705652    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.706425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.706245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.706529 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.706500    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.707794 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.707772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.710885 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.709923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.710885 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.710401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.711576    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.711747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.712186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.713104    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.713189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.714807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.714947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.714881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.715381 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.715009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.716339 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.715808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s22d\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.717488 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.717237    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:21.855164 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:21.855122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:24.660032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:24.659898    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:39:24.661128 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:39:24.661099    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6486060_5539_477c_bd54_975b649b8c08.slice/crio-a7d9bb92b71f24dd82a626e68018150a0d680e5af2b9ac7bfd822ed29b09751d WatchSource:0}: Error finding container a7d9bb92b71f24dd82a626e68018150a0d680e5af2b9ac7bfd822ed29b09751d: Status 404 returned error can't find the container with id a7d9bb92b71f24dd82a626e68018150a0d680e5af2b9ac7bfd822ed29b09751d
Sep 18 19:39:25.302522 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.302358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kh7cd" event={"ID":"52504afc-712e-48f5-a8d4-586460e326c5","Type":"ContainerStarted","Data":"240c90187934c57c6448c8706df09b18599178c16758501a4f809c6f5f062cc4"}
Sep 18 19:39:25.302522 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.302479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:39:25.304766 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.304722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f75sh" event={"ID":"a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d","Type":"ContainerStarted","Data":"93b33567f58f75f4562977b81ec591d6284bd1368315cf9883a05260c3166a1f"}
Sep 18 19:39:25.307905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.306985    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81" exitCode=0
Sep 18 19:39:25.307905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.307068    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"}
Sep 18 19:39:25.307905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.307131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"a7d9bb92b71f24dd82a626e68018150a0d680e5af2b9ac7bfd822ed29b09751d"}
Sep 18 19:39:25.312795 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.312492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"}
Sep 18 19:39:25.312795 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.312607    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"}
Sep 18 19:39:25.312795 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.312671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"}
Sep 18 19:39:25.312795 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.312714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"}
Sep 18 19:39:25.312795 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.312751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"}
Sep 18 19:39:25.315977 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.315954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"fbc5e00e501308a60c25c4b599d5884bd2e617aa22f89e6d0b11bad451020ced"}
Sep 18 19:39:25.316074 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.316020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"2fbc247004e7dd09ade86ae55c7e64fadcca003c079030e535cacd02edc55701"}
Sep 18 19:39:25.316074 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.316061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"351c4dda40c76d8ed2eb2a89dbcc49e3e05dfd073997cbcf57b1c6115b340e02"}
Sep 18 19:39:25.318616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.317754    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kh7cd" podStartSLOduration=67.21398233 podStartE2EDuration="1m12.317736785s" totalImagesPullingTime="5.103754455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:25.316693422 +0000 UTC m=+73.834663063" watchObservedRunningTime="2026-09-18 19:39:25.317736785 +0000 UTC m=+73.835706427"
Sep 18 19:39:25.319026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.319003    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8grvn" event={"ID":"87591871-d05d-4e65-ba24-915d085bf6b2","Type":"ContainerStarted","Data":"4aafd8aae32881455e2586b6f16a6c74429b1da444f645164123b382f55fa781"}
Sep 18 19:39:25.319100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.319078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8grvn" event={"ID":"87591871-d05d-4e65-ba24-915d085bf6b2","Type":"ContainerStarted","Data":"da5ff0caff10fb7f335696933f2c63cb2fef13898e55cc7905e9d0078acfb12f"}
Sep 18 19:39:25.357023 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.356980    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f75sh" podStartSLOduration=67.241011013 podStartE2EDuration="1m12.356966882s" totalImagesPullingTime="5.115955869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:25.355543804 +0000 UTC m=+73.873513446" watchObservedRunningTime="2026-09-18 19:39:25.356966882 +0000 UTC m=+73.874936521"
Sep 18 19:39:25.375881 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:25.375835    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8grvn" podStartSLOduration=67.296496155 podStartE2EDuration="1m12.375822194s" totalImagesPullingTime="5.079326039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:25.37470469 +0000 UTC m=+73.892674355" watchObservedRunningTime="2026-09-18 19:39:25.375822194 +0000 UTC m=+73.893791834"
Sep 18 19:39:26.127483 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.127456    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:39:26.329142 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.329090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerStarted","Data":"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"}
Sep 18 19:39:26.332980 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.332875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"74f332c395f84de763276bc15f5b973c3bb4da71658b09d88b4fab71692c1068"}
Sep 18 19:39:26.332980 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.332952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"e9972e186038a7cbca5f8a891d609f27c757afdaa40b33171c5ff63f52466a99"}
Sep 18 19:39:26.333174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.332991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" event={"ID":"e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0","Type":"ContainerStarted","Data":"69b5df3f33a15fc1a8aeaf0f94e6b026b5000685b481a6da661dae96ba8e0b69"}
Sep 18 19:39:26.333671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.333601    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:26.355608 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.355555    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.1846272190000002 podStartE2EDuration="10.355536164s" totalImagesPullingTime="7.141622523s" totalInitContainerRuntime="29.286422ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:26.353740242 +0000 UTC m=+74.871709886" watchObservedRunningTime="2026-09-18 19:39:26.355536164 +0000 UTC m=+74.873505807"
Sep 18 19:39:26.376808 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:26.376767    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx" podStartSLOduration=2.564002013 podStartE2EDuration="9.376751382s" totalImagesPullingTime="6.812749369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:26.375596719 +0000 UTC m=+74.893566362" watchObservedRunningTime="2026-09-18 19:39:26.376751382 +0000 UTC m=+74.894721021"
Sep 18 19:39:28.343791 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:28.343765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"}
Sep 18 19:39:29.350604 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.350561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"}
Sep 18 19:39:29.350945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.350655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"}
Sep 18 19:39:29.350945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.350680    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"}
Sep 18 19:39:29.350945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.350704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"}
Sep 18 19:39:29.350945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.350726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerStarted","Data":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"}
Sep 18 19:39:29.382502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:29.382449    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.420077468 podStartE2EDuration="8.382431603s" totalImagesPullingTime="2.933467393s" totalInitContainerRuntime="28.886742ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:29.380002593 +0000 UTC m=+77.897972235" watchObservedRunningTime="2026-09-18 19:39:29.382431603 +0000 UTC m=+77.900401243"
Sep 18 19:39:31.855957 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:31.855913    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:39:32.343526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:32.343501    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55c6b56b78-5q7rx"
Sep 18 19:39:41.140266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.140185    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9698db596-t5mrz" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" containerName="registry" containerID="cri-o://3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3" gracePeriod=30
Sep 18 19:39:41.371299 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.371277    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:39:41.397430 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.397366    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dbc201f-a801-49fc-a806-66bcd971853d" containerID="3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3" exitCode=0
Sep 18 19:39:41.397430 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.397402    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9698db596-t5mrz"
Sep 18 19:39:41.397596 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.397449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9698db596-t5mrz" event={"ID":"0dbc201f-a801-49fc-a806-66bcd971853d","Type":"ContainerDied","Data":"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"}
Sep 18 19:39:41.397596 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.397503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9698db596-t5mrz" event={"ID":"0dbc201f-a801-49fc-a806-66bcd971853d","Type":"ContainerDied","Data":"b512f295f7ee00e09b087f3087143fc868bec73cf27b204bb01a55847811daaf"}
Sep 18 19:39:41.397596 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.397533    2573 scope.go:117] "RemoveContainer" containerID="3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"
Sep 18 19:39:41.405698 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.405682    2573 scope.go:117] "RemoveContainer" containerID="3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"
Sep 18 19:39:41.406006 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:39:41.405983    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3\": container with ID starting with 3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3 not found: ID does not exist" containerID="3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"
Sep 18 19:39:41.406088 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.406012    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3"} err="failed to get container status \"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3\": rpc error: code = NotFound desc = could not find container \"3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3\": container with ID starting with 3bca14ea1ae57b25b031df4dd576be446190e3142c1ab40f1d4a04c6fb2821e3 not found: ID does not exist"
Sep 18 19:39:41.471760 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.471740    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.471869 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.471771    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxn7s\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.471869 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.471798    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.471981 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.471948    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.472036 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.471993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.472088 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.472053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.472150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.472104    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.472150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.472132    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted\") pod \"0dbc201f-a801-49fc-a806-66bcd971853d\" (UID: \"0dbc201f-a801-49fc-a806-66bcd971853d\") "
Sep 18 19:39:41.472483 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.472456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:41.472643 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.472526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:39:41.474425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.474395    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:41.474521 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.474490    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:39:41.474521 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.474494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s" (OuterVolumeSpecName: "kube-api-access-qxn7s") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "kube-api-access-qxn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:41.474746 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.474729    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:41.474822 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.474802    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:39:41.481633 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.481607    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0dbc201f-a801-49fc-a806-66bcd971853d" (UID: "0dbc201f-a801-49fc-a806-66bcd971853d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:39:41.573411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573372    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-bound-sa-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573398    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dbc201f-a801-49fc-a806-66bcd971853d-ca-trust-extracted\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573408    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-installation-pull-secrets\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573417    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn7s\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-kube-api-access-qxn7s\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573426    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dbc201f-a801-49fc-a806-66bcd971853d-registry-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573437    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dbc201f-a801-49fc-a806-66bcd971853d-image-registry-private-configuration\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573446    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-registry-certificates\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.573743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.573456    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dbc201f-a801-49fc-a806-66bcd971853d-trusted-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:39:41.716485 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.716461    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:39:41.720250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.720227    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9698db596-t5mrz"]
Sep 18 19:39:41.782195 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:41.782166    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" path="/var/lib/kubelet/pods/0dbc201f-a801-49fc-a806-66bcd971853d/volumes"
Sep 18 19:39:56.335332 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:39:56.335300    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kh7cd"
Sep 18 19:40:21.855904 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:21.855864    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:21.874822 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:21.874796    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:22.563503 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:22.563474    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:35.634570 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.634527    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:35.635162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635017    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="alertmanager" containerID="cri-o://653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e" gracePeriod=120
Sep 18 19:40:35.635162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635101    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-metric" containerID="cri-o://0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23" gracePeriod=120
Sep 18 19:40:35.635162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635129    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="config-reloader" containerID="cri-o://f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b" gracePeriod=120
Sep 18 19:40:35.635366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635154    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy" containerID="cri-o://8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1" gracePeriod=120
Sep 18 19:40:35.635366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635111    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-web" containerID="cri-o://9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a" gracePeriod=120
Sep 18 19:40:35.635366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:35.635309    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="prom-label-proxy" containerID="cri-o://6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d" gracePeriod=120
Sep 18 19:40:36.600339 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600304    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d" exitCode=0
Sep 18 19:40:36.600339 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600331    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1" exitCode=0
Sep 18 19:40:36.600339 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600341    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b" exitCode=0
Sep 18 19:40:36.600339 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600348    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e" exitCode=0
Sep 18 19:40:36.600614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"}
Sep 18 19:40:36.600614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600421    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"}
Sep 18 19:40:36.600614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"}
Sep 18 19:40:36.600614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.600475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"}
Sep 18 19:40:36.875398 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.875371    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:36.992419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992388    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992436    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkdql\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992772 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992656    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992772 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992705    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992772 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992734    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992804    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992839    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.992926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992916    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.993165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.992948    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle\") pod \"d46b3a1b-e45c-4823-8c53-db423ffa9570\" (UID: \"d46b3a1b-e45c-4823-8c53-db423ffa9570\") "
Sep 18 19:40:36.993607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.993310    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:36.993607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.993533    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:36.994694 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.994661    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:36.996078 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996047    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql" (OuterVolumeSpecName: "kube-api-access-lkdql") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "kube-api-access-lkdql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:36.996176 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996071    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:36.996269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996221    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:36.996368 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996347    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out" (OuterVolumeSpecName: "config-out") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:36.996677 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996653    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:36.996999 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.996975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:36.997123 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.997100    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:36.997373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:36.997356    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume" (OuterVolumeSpecName: "config-volume") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:37.000880 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.000789    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:37.006099 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.006077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config" (OuterVolumeSpecName: "web-config") pod "d46b3a1b-e45c-4823-8c53-db423ffa9570" (UID: "d46b3a1b-e45c-4823-8c53-db423ffa9570"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:37.094422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094390    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-main-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094416    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdql\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-kube-api-access-lkdql\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094425    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d46b3a1b-e45c-4823-8c53-db423ffa9570-tls-assets\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094435    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094445    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-web-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094454    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-main-db\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094462    2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-volume\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094470    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-metrics-client-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094478    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094487    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094495    2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d46b3a1b-e45c-4823-8c53-db423ffa9570-cluster-tls-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094503    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d46b3a1b-e45c-4823-8c53-db423ffa9570-config-out\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.094651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.094511    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d46b3a1b-e45c-4823-8c53-db423ffa9570-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:37.606851 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606814    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23" exitCode=0
Sep 18 19:40:37.606851 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606841    2573 generic.go:356] "Generic (PLEG): container finished" podID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerID="9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a" exitCode=0
Sep 18 19:40:37.607045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"}
Sep 18 19:40:37.607045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606893    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.607045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"}
Sep 18 19:40:37.607045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d46b3a1b-e45c-4823-8c53-db423ffa9570","Type":"ContainerDied","Data":"51077128615ae4503d1c317fac74dd75084b379758f175c25ecd3ada7d25b612"}
Sep 18 19:40:37.607045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.606984    2573 scope.go:117] "RemoveContainer" containerID="6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"
Sep 18 19:40:37.614560 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.614536    2573 scope.go:117] "RemoveContainer" containerID="0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"
Sep 18 19:40:37.621655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.621640    2573 scope.go:117] "RemoveContainer" containerID="8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"
Sep 18 19:40:37.628120 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.628080    2573 scope.go:117] "RemoveContainer" containerID="9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"
Sep 18 19:40:37.631521 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.631490    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:37.635415 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.635394    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:37.636579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.636565    2573 scope.go:117] "RemoveContainer" containerID="f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"
Sep 18 19:40:37.643425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.643405    2573 scope.go:117] "RemoveContainer" containerID="653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"
Sep 18 19:40:37.650389 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.650373    2573 scope.go:117] "RemoveContainer" containerID="829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"
Sep 18 19:40:37.657236 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657218    2573 scope.go:117] "RemoveContainer" containerID="6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"
Sep 18 19:40:37.657565 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.657544    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d\": container with ID starting with 6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d not found: ID does not exist" containerID="6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"
Sep 18 19:40:37.657619 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657573    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"} err="failed to get container status \"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d\": rpc error: code = NotFound desc = could not find container \"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d\": container with ID starting with 6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d not found: ID does not exist"
Sep 18 19:40:37.657619 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657593    2573 scope.go:117] "RemoveContainer" containerID="0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"
Sep 18 19:40:37.657751 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657643    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:37.657853 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.657834    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23\": container with ID starting with 0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23 not found: ID does not exist" containerID="0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"
Sep 18 19:40:37.657891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657860    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"} err="failed to get container status \"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23\": rpc error: code = NotFound desc = could not find container \"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23\": container with ID starting with 0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23 not found: ID does not exist"
Sep 18 19:40:37.657891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657876    2573 scope.go:117] "RemoveContainer" containerID="8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"
Sep 18 19:40:37.657962 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657914    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="init-config-reloader"
Sep 18 19:40:37.657962 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657928    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="init-config-reloader"
Sep 18 19:40:37.657962 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657951    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="alertmanager"
Sep 18 19:40:37.657962 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657960    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="alertmanager"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657970    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="prom-label-proxy"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657979    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="prom-label-proxy"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657988    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-metric"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.657996    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-metric"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658008    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" containerName="registry"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658016    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" containerName="registry"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658025    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658033    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658043    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-web"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658052    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-web"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658065    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="config-reloader"
Sep 18 19:40:37.658098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658074    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="config-reloader"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.658111    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1\": container with ID starting with 8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1 not found: ID does not exist" containerID="8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658129    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658127    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"} err="failed to get container status \"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1\": rpc error: code = NotFound desc = could not find container \"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1\": container with ID starting with 8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1 not found: ID does not exist"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658139    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="config-reloader"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658147    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="prom-label-proxy"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658153    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-web"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658158    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="kube-rbac-proxy-metric"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658164    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbc201f-a801-49fc-a806-66bcd971853d" containerName="registry"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658170    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" containerName="alertmanager"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658140    2573 scope.go:117] "RemoveContainer" containerID="9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.658468    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a\": container with ID starting with 9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a not found: ID does not exist" containerID="9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"
Sep 18 19:40:37.658502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658497    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"} err="failed to get container status \"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a\": rpc error: code = NotFound desc = could not find container \"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a\": container with ID starting with 9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a not found: ID does not exist"
Sep 18 19:40:37.658923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658517    2573 scope.go:117] "RemoveContainer" containerID="f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"
Sep 18 19:40:37.658923 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.658763    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b\": container with ID starting with f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b not found: ID does not exist" containerID="f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"
Sep 18 19:40:37.658923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658782    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"} err="failed to get container status \"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b\": rpc error: code = NotFound desc = could not find container \"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b\": container with ID starting with f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b not found: ID does not exist"
Sep 18 19:40:37.658923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.658795    2573 scope.go:117] "RemoveContainer" containerID="653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"
Sep 18 19:40:37.659069 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.659051    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e\": container with ID starting with 653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e not found: ID does not exist" containerID="653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"
Sep 18 19:40:37.659110 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659078    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"} err="failed to get container status \"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e\": rpc error: code = NotFound desc = could not find container \"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e\": container with ID starting with 653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e not found: ID does not exist"
Sep 18 19:40:37.659110 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659093    2573 scope.go:117] "RemoveContainer" containerID="829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"
Sep 18 19:40:37.659368 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:37.659340    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac\": container with ID starting with 829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac not found: ID does not exist" containerID="829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"
Sep 18 19:40:37.659445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659364    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"} err="failed to get container status \"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac\": rpc error: code = NotFound desc = could not find container \"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac\": container with ID starting with 829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac not found: ID does not exist"
Sep 18 19:40:37.659445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659383    2573 scope.go:117] "RemoveContainer" containerID="6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"
Sep 18 19:40:37.659607 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659591    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d"} err="failed to get container status \"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d\": rpc error: code = NotFound desc = could not find container \"6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d\": container with ID starting with 6b56e0aa1daf640b914a9d58b97cd33f2b5cae0e9ec86a13bab24ed96fc8259d not found: ID does not exist"
Sep 18 19:40:37.659650 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659607    2573 scope.go:117] "RemoveContainer" containerID="0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"
Sep 18 19:40:37.659830 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659811    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23"} err="failed to get container status \"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23\": rpc error: code = NotFound desc = could not find container \"0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23\": container with ID starting with 0c58bed7e80a7eaece84a03c9823cfc8c0ed6a20a8e0b9a2dcce92edb525ca23 not found: ID does not exist"
Sep 18 19:40:37.659908 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.659831    2573 scope.go:117] "RemoveContainer" containerID="8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"
Sep 18 19:40:37.660102 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660083    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1"} err="failed to get container status \"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1\": rpc error: code = NotFound desc = could not find container \"8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1\": container with ID starting with 8abc439aea990f794804e847ac91f1b3d72806f7c146b84bc4cc7cfcfd2184f1 not found: ID does not exist"
Sep 18 19:40:37.660102 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660101    2573 scope.go:117] "RemoveContainer" containerID="9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"
Sep 18 19:40:37.660398 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660381    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a"} err="failed to get container status \"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a\": rpc error: code = NotFound desc = could not find container \"9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a\": container with ID starting with 9545ec124bfd91c321204cfbfb4744cf63ee2e6bebe4d7566721ef4f62a1f05a not found: ID does not exist"
Sep 18 19:40:37.660459 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660398    2573 scope.go:117] "RemoveContainer" containerID="f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"
Sep 18 19:40:37.660649 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660631    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b"} err="failed to get container status \"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b\": rpc error: code = NotFound desc = could not find container \"f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b\": container with ID starting with f330053ea34f1a98446ae9aa7fb4e6852913a71e170cb270c26041d78c931a5b not found: ID does not exist"
Sep 18 19:40:37.660690 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660650    2573 scope.go:117] "RemoveContainer" containerID="653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"
Sep 18 19:40:37.660880 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660862    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e"} err="failed to get container status \"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e\": rpc error: code = NotFound desc = could not find container \"653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e\": container with ID starting with 653b3942645beb349cbd9aa452bc59aad5dae9a179c9144255cb393bf5ae6b6e not found: ID does not exist"
Sep 18 19:40:37.660922 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.660879    2573 scope.go:117] "RemoveContainer" containerID="829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"
Sep 18 19:40:37.661099 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.661083    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac"} err="failed to get container status \"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac\": rpc error: code = NotFound desc = could not find container \"829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac\": container with ID starting with 829dd9752491fa1a06a001cc06d84de3a8c6e4b5ff7f752e51b99713f011bcac not found: ID does not exist"
Sep 18 19:40:37.661960 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.661947    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.664257 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664236    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:40:37.664403 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664383    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rkpdl\""
Sep 18 19:40:37.664508 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664411    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:40:37.664508 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664478    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:40:37.664665 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664648    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:40:37.664722 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664674    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:40:37.664782 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664746    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:40:37.664883 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.664866    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:40:37.665032 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.665012    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:40:37.672716 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.672689    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:40:37.676807 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.676787    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:37.698090 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-web-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698267    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgn59\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-kube-api-access-hgn59\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-out\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.698449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.698440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.782302 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.782269    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d46b3a1b-e45c-4823-8c53-db423ffa9570" path="/var/lib/kubelet/pods/d46b3a1b-e45c-4823-8c53-db423ffa9570/volumes"
Sep 18 19:40:37.799680 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.799840 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.799840 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.799905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-web-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.799905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.799905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799891    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.799971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.800008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.800040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.800065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgn59\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-kube-api-access-hgn59\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.800095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-out\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.800174 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.800117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.801543 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.801486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.801718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.801692    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.801831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.801699    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d743314e-1d70-4be3-9445-b2c9d9280bb2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803009 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.802984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803161 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803135    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-web-config\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803245 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803361 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803696 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-volume\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803696 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.803802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.803696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d743314e-1d70-4be3-9445-b2c9d9280bb2-config-out\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.804620 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.804603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.804801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.804783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d743314e-1d70-4be3-9445-b2c9d9280bb2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.807929 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.807912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgn59\" (UniqueName: \"kubernetes.io/projected/d743314e-1d70-4be3-9445-b2c9d9280bb2-kube-api-access-hgn59\") pod \"alertmanager-main-0\" (UID: \"d743314e-1d70-4be3-9445-b2c9d9280bb2\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:37.973430 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:37.973323    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:40:38.251949 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:38.251924    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:40:38.253794 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:40:38.253768    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd743314e_1d70_4be3_9445_b2c9d9280bb2.slice/crio-2e31f090074f38174fbc1f5b4891acdb6b7e22af2cc03c0d99d42f0b6b4f8d12 WatchSource:0}: Error finding container 2e31f090074f38174fbc1f5b4891acdb6b7e22af2cc03c0d99d42f0b6b4f8d12: Status 404 returned error can't find the container with id 2e31f090074f38174fbc1f5b4891acdb6b7e22af2cc03c0d99d42f0b6b4f8d12
Sep 18 19:40:38.611702 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:38.611672    2573 generic.go:356] "Generic (PLEG): container finished" podID="d743314e-1d70-4be3-9445-b2c9d9280bb2" containerID="5f025666b0630aa7b4394b8d54bdfbf6dd3c7645c9dcc8e66ae651a9e128e1bd" exitCode=0
Sep 18 19:40:38.611868 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:38.611765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerDied","Data":"5f025666b0630aa7b4394b8d54bdfbf6dd3c7645c9dcc8e66ae651a9e128e1bd"}
Sep 18 19:40:38.611868 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:38.611837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"2e31f090074f38174fbc1f5b4891acdb6b7e22af2cc03c0d99d42f0b6b4f8d12"}
Sep 18 19:40:39.625422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"7b4fb664ffa506bfcee6adf9e41f2e43dae49e293fa6414ba358d1bea7b3a133"}
Sep 18 19:40:39.625914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"21e0d96200ccc66ab940308823ce10bba9e1da3c8519c6b66a63ba4791e57bf8"}
Sep 18 19:40:39.625914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625535    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"92b36d16ed968e2313032ad6798d6416f7fe807b1fd598bf2a46c42235ab59ab"}
Sep 18 19:40:39.625914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"55282ae23f3f3b7abb71bf44f9436bf8739cede4b14608d71f5cdf8602f5e962"}
Sep 18 19:40:39.625914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"3b69918830fcdebe7f08d954f4b4e7506beadb6af5c90c8306d456e65a534a0a"}
Sep 18 19:40:39.625914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.625664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d743314e-1d70-4be3-9445-b2c9d9280bb2","Type":"ContainerStarted","Data":"0285641044f4c78b21962598ea9ed3458620a4b3cba41ffd69d9eea281466415"}
Sep 18 19:40:39.651281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.651225    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.63385432 podStartE2EDuration="2.651189185s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.334865ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:39.650077589 +0000 UTC m=+148.168047241" watchObservedRunningTime="2026-09-18 19:40:39.651189185 +0000 UTC m=+148.169158827"
Sep 18 19:40:39.861218 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861166    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:39.861544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861513    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="prometheus" containerID="cri-o://98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955" gracePeriod=600
Sep 18 19:40:39.861692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861535    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy" containerID="cri-o://93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e" gracePeriod=600
Sep 18 19:40:39.861692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861567    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277" gracePeriod=600
Sep 18 19:40:39.861692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861595    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-web" containerID="cri-o://e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409" gracePeriod=600
Sep 18 19:40:39.861692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861618    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="config-reloader" containerID="cri-o://5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f" gracePeriod=600
Sep 18 19:40:39.861692 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:39.861573    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="thanos-sidecar" containerID="cri-o://7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91" gracePeriod=600
Sep 18 19:40:40.099995 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.099971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.219997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.219899    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.219997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.219953    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.219997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.219979    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s22d\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220022    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220042    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220061    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220076    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220095    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220115    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220140    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220167    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220192    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220255    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220285    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220336    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220406    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220432    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220456    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out\") pod \"c6486060-5539-477c-bd54-975b649b8c08\" (UID: \"c6486060-5539-477c-bd54-975b649b8c08\") "
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220509    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.220866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.220705    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.221471 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.221444    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.221822 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.221798    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.222414 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.222386    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.222512 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.222472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:40.223021 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.222979    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.223278 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.223253    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.223380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.223283    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:40.223483 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.223461    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.224740 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.224710    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d" (OuterVolumeSpecName: "kube-api-access-4s22d") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "kube-api-access-4s22d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:40.224842 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.224751    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.224910 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.224872    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:40.224992 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.224967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.225090 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225052    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.225217 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225134    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.225294 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225257    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config" (OuterVolumeSpecName: "config") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.225370 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225348    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out" (OuterVolumeSpecName: "config-out") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:40.225490 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.225957 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.225940    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.234033 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.234011    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config" (OuterVolumeSpecName: "web-config") pod "c6486060-5539-477c-bd54-975b649b8c08" (UID: "c6486060-5539-477c-bd54-975b649b8c08"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:40.321111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321072    2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-metrics-client-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321103    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321114    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s22d\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-kube-api-access-4s22d\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321123    2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-kube-rbac-proxy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321132    2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321141    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321150    2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-grpc-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321158    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321166    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-metrics-client-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321176    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321184    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321193    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321222    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-db\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321230    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6486060-5539-477c-bd54-975b649b8c08-tls-assets\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321238    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321246    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-web-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321254    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c6486060-5539-477c-bd54-975b649b8c08-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321264    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6486060-5539-477c-bd54-975b649b8c08-config-out\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.321380 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.321272    2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c6486060-5539-477c-bd54-975b649b8c08-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:40:40.632226 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632181    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277" exitCode=0
Sep 18 19:40:40.632226 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632220    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e" exitCode=0
Sep 18 19:40:40.632226 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632229    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409" exitCode=0
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632238    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91" exitCode=0
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632245    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f" exitCode=0
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632322    2573 generic.go:356] "Generic (PLEG): container finished" podID="c6486060-5539-477c-bd54-975b649b8c08" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955" exitCode=0
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632390    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"}
Sep 18 19:40:40.632774 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.632466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c6486060-5539-477c-bd54-975b649b8c08","Type":"ContainerDied","Data":"a7d9bb92b71f24dd82a626e68018150a0d680e5af2b9ac7bfd822ed29b09751d"}
Sep 18 19:40:40.643108 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.643088    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.650573 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.650557    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.656325 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.656305    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:40.658429 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.658376    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.659709 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.659692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:40.664997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.664984    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.671606 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.671587    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.678604 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.678586    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.683513 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683490    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:40.683767 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683752    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="prometheus"
Sep 18 19:40:40.683767 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683765    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="prometheus"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683773    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="thanos-sidecar"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683779    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="thanos-sidecar"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683793    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="init-config-reloader"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683798    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="init-config-reloader"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683805    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-thanos"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683810    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-thanos"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683818    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="config-reloader"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683823    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="config-reloader"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683830    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683835    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683842    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-web"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683847    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-web"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683893    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="config-reloader"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683901    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-web"
Sep 18 19:40:40.683902 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683908    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy-thanos"
Sep 18 19:40:40.684439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683914    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="kube-rbac-proxy"
Sep 18 19:40:40.684439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683922    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="prometheus"
Sep 18 19:40:40.684439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.683927    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6486060-5539-477c-bd54-975b649b8c08" containerName="thanos-sidecar"
Sep 18 19:40:40.687827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.687809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.691069 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.691054    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:40:40.691409 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.691395    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:40:40.691846 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.691831    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9sfxp\""
Sep 18 19:40:40.691969 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.691955    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:40:40.693385 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.693364    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:40:40.693662 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.693644    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-63drttkr3886n\""
Sep 18 19:40:40.693909 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.693892    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:40:40.694415 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.694399    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:40:40.694518 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.694426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:40:40.694518 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.694456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:40:40.696801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.696781    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.697075 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.697048    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.697143 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697076    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.697143 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697096    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.697409 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.697392    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.697460 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697417    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.697460 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697433    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.697703 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.697685    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.697775 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697704    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.697775 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697716    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.697935 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.697918    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.697989 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697939    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.697989 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.697953    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.698177 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.698161    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.698266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698179    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.698266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698191    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.698519 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.698479    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.698596 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698526    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.698596 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698542    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.698819 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:40:40.698802    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.698860 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698826    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.698860 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.698843    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.699064 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699047    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.699064 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699064    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.699349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699321    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.699489 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699351    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.699718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699639    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.699718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699661    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.700006 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.699981    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.700090 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700009    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.700301 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700284    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:40:40.700356 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700282    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.700356 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700321    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.700585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700566    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.700633 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700586    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.700840 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700822    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.700908 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.700842    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.701080 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701062    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.701137 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701080    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.701350 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701332    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.701402 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701350    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.701579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701558    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.701579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701578    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.701814 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701797    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.701872 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.701815    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.702044 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702023    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.702084 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702044    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.702276 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702259    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.702353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702278    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.702513 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702497    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.702560 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702514    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.702707 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702690    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.702776 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702709    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.702943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702924    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.702985 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.702944    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.703123 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703108    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.703168 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703122    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.703365 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703349    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.703427 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703366    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.703551 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703535    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.703611 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703555    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.703796 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703781    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.703838 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703797    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.704014 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.703996    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.704057 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704015    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.704297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704279    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.704372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704301    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.704523 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704507    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.704570 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704524    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.704714 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704699    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.704714 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704712    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.704946 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704927    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.704984 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.704947    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.705151 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705137    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.705188 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705152    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.705376 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705356    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.705411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705377    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.705587 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705566    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.705626 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705588    2573 scope.go:117] "RemoveContainer" containerID="2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"
Sep 18 19:40:40.705786 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705771    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277"} err="failed to get container status \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": rpc error: code = NotFound desc = could not find container \"2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277\": container with ID starting with 2f96d41bb8a18437d779de8aee5222ea98b43d4e8f07d4e9363596a9d3bbf277 not found: ID does not exist"
Sep 18 19:40:40.705819 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705786    2573 scope.go:117] "RemoveContainer" containerID="93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"
Sep 18 19:40:40.705995 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705980    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e"} err="failed to get container status \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": rpc error: code = NotFound desc = could not find container \"93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e\": container with ID starting with 93ec68768e8607ee8fd7a122b6fabca55c2f04d23528e4efa34f388ee24d429e not found: ID does not exist"
Sep 18 19:40:40.706028 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.705996    2573 scope.go:117] "RemoveContainer" containerID="e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"
Sep 18 19:40:40.706210 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706180    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409"} err="failed to get container status \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": rpc error: code = NotFound desc = could not find container \"e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409\": container with ID starting with e878d7606d3f196c7d9d2a3912b5a81f48d657cd1c6b4101163f9e6ea1325409 not found: ID does not exist"
Sep 18 19:40:40.706269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706256    2573 scope.go:117] "RemoveContainer" containerID="7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"
Sep 18 19:40:40.706472 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706456    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91"} err="failed to get container status \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": rpc error: code = NotFound desc = could not find container \"7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91\": container with ID starting with 7353532a6be0fd11fb03a66d10473ab7b8aef7199db43805da04414f8b8ddc91 not found: ID does not exist"
Sep 18 19:40:40.706505 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706474    2573 scope.go:117] "RemoveContainer" containerID="5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"
Sep 18 19:40:40.706694 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706668    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f"} err="failed to get container status \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": rpc error: code = NotFound desc = could not find container \"5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f\": container with ID starting with 5a29b9bde73cbda2265b132981034fca668433e57f5274412d60d1dfa619f89f not found: ID does not exist"
Sep 18 19:40:40.706728 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706696    2573 scope.go:117] "RemoveContainer" containerID="98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"
Sep 18 19:40:40.706901 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706884    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955"} err="failed to get container status \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": rpc error: code = NotFound desc = could not find container \"98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955\": container with ID starting with 98d1730db2f7a9b2ffc7e9f4148f9798b28bd826ab28b5b47512b61baeae6955 not found: ID does not exist"
Sep 18 19:40:40.706936 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.706901    2573 scope.go:117] "RemoveContainer" containerID="198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"
Sep 18 19:40:40.707110 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.707094    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81"} err="failed to get container status \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": rpc error: code = NotFound desc = could not find container \"198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81\": container with ID starting with 198b24b5bd694317dee4fa42c6b79a4f8eec34de82166d5c5d46c07c818bcb81 not found: ID does not exist"
Sep 18 19:40:40.712309 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.712293    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:40:40.712448 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.712432    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:40:40.713055 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.712939    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:40:40.713584 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.713385    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:40:40.718509 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.718482    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:40:40.724727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.724801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724734    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9854r\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-kube-api-access-9854r\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.724801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.724913 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.724913 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.724959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725088 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725088 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725088 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725243 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725243 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725243 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725238    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725367 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725367 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725468 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725468 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725468 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.725579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.725544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.733706 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.733683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:40.736985 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.736964    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:40:40.825924 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.825888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.825924 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.825929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826181 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.825953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826181 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.825976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826181 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826181 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826154    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9854r\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-kube-api-access-9854r\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826546    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.826727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.827145 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.827145 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.826935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.827635 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.827609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.829533 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.829533 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829194    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.829533 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829391    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830002 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829630    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830002 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829829    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830002 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.829853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830157 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.830071    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830316 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.830294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.830403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.830804 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.830774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.831587 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.831564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.831663 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.831608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e91a6d-98ca-453f-800d-a8f36f39518a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.832386 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.832367    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.832474 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.832458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e91a6d-98ca-453f-800d-a8f36f39518a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.832831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.832811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e91a6d-98ca-453f-800d-a8f36f39518a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:40.837538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:40.837517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9854r\" (UniqueName: \"kubernetes.io/projected/f9e91a6d-98ca-453f-800d-a8f36f39518a-kube-api-access-9854r\") pod \"prometheus-k8s-0\" (UID: \"f9e91a6d-98ca-453f-800d-a8f36f39518a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:41.002511 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.002418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:40:41.281866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.281841    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:40:41.284216 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:40:41.284172    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e91a6d_98ca_453f_800d_a8f36f39518a.slice/crio-6b990e9506c572bead41e4ac92ab562728a905c3a66499675f75262705e6c2f8 WatchSource:0}: Error finding container 6b990e9506c572bead41e4ac92ab562728a905c3a66499675f75262705e6c2f8: Status 404 returned error can't find the container with id 6b990e9506c572bead41e4ac92ab562728a905c3a66499675f75262705e6c2f8
Sep 18 19:40:41.638943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.638902    2573 generic.go:356] "Generic (PLEG): container finished" podID="f9e91a6d-98ca-453f-800d-a8f36f39518a" containerID="fcc048997e772ed4a4a2ff79482bfca7f0b6d071ef67202d93b3daf8b830719c" exitCode=0
Sep 18 19:40:41.639373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.638988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerDied","Data":"fcc048997e772ed4a4a2ff79482bfca7f0b6d071ef67202d93b3daf8b830719c"}
Sep 18 19:40:41.639373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.639069    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"6b990e9506c572bead41e4ac92ab562728a905c3a66499675f75262705e6c2f8"}
Sep 18 19:40:41.783162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:41.783134    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6486060-5539-477c-bd54-975b649b8c08" path="/var/lib/kubelet/pods/c6486060-5539-477c-bd54-975b649b8c08/volumes"
Sep 18 19:40:42.646965 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.646925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"701c13cea44a33b94852f1cd0920c95f812d84aff8e60d3fb720cd63e2dd2c35"}
Sep 18 19:40:42.647491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.647041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"ece22f8213d3787e45446e70e969523e736765a17160e8c778c82c20540cfa5d"}
Sep 18 19:40:42.647491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.647089    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"e5a4500601886fc16a5240bb714f160ae17b32324860d55499d128ca2a01b9c1"}
Sep 18 19:40:42.647491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.647134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"7aca917643d4796d1fe776ae5f3624343281cbacddbf24eb10cfcf7d13f4fd22"}
Sep 18 19:40:42.647491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.647172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"c8270f28f891f34513681c76b27e7f16aa36429cd94358f9463dbc03287d4183"}
Sep 18 19:40:42.647491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.647235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e91a6d-98ca-453f-800d-a8f36f39518a","Type":"ContainerStarted","Data":"37234c9e2a24cb2648f72975bb24cb5320b49099d3becea773ccf469ae91e7e4"}
Sep 18 19:40:42.676331 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:42.676268    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.656380064 podStartE2EDuration="2.676247435s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.867371ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:42.674901593 +0000 UTC m=+151.192871233" watchObservedRunningTime="2026-09-18 19:40:42.676247435 +0000 UTC m=+151.194217076"
Sep 18 19:40:46.003538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:40:46.003484    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:41.003577 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:41:41.003537    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:41.023518 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:41:41.023489    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:41:41.882439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:41:41.882411    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:43:13.695180 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:13.695155    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:21.155036 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.154992    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2ghw8"]
Sep 18 19:43:21.156926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.156911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.159683 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.159664    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xnn8t\""
Sep 18 19:43:21.159683 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.159667    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:43:21.159905 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.159889    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:43:21.160484 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.160466    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:43:21.174005 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.173974    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ghw8"]
Sep 18 19:43:21.288185 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.288149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbeea091-d888-4b30-a394-30edd3e366c4-data\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.288407 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.288270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5sbj\" (UniqueName: \"kubernetes.io/projected/cbeea091-d888-4b30-a394-30edd3e366c4-kube-api-access-r5sbj\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.389537 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.389494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbeea091-d888-4b30-a394-30edd3e366c4-data\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.389711 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.389565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5sbj\" (UniqueName: \"kubernetes.io/projected/cbeea091-d888-4b30-a394-30edd3e366c4-kube-api-access-r5sbj\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.389935 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.389912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbeea091-d888-4b30-a394-30edd3e366c4-data\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.398591 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.398557    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5sbj\" (UniqueName: \"kubernetes.io/projected/cbeea091-d888-4b30-a394-30edd3e366c4-kube-api-access-r5sbj\") pod \"seaweedfs-64568bcd49-2ghw8\" (UID: \"cbeea091-d888-4b30-a394-30edd3e366c4\") " pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.465516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.465435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:21.732697 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.732626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2ghw8"]
Sep 18 19:43:21.735789 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:43:21.735759    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbeea091_d888_4b30_a394_30edd3e366c4.slice/crio-edfb66d16bd363b3bd5983f9e4d18eba0d3d0bdef907f2665104ce977e933c8a WatchSource:0}: Error finding container edfb66d16bd363b3bd5983f9e4d18eba0d3d0bdef907f2665104ce977e933c8a: Status 404 returned error can't find the container with id edfb66d16bd363b3bd5983f9e4d18eba0d3d0bdef907f2665104ce977e933c8a
Sep 18 19:43:21.737050 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:21.737034    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:22.231817 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:22.231776    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ghw8" event={"ID":"cbeea091-d888-4b30-a394-30edd3e366c4","Type":"ContainerStarted","Data":"edfb66d16bd363b3bd5983f9e4d18eba0d3d0bdef907f2665104ce977e933c8a"}
Sep 18 19:43:25.246066 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:25.246024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2ghw8" event={"ID":"cbeea091-d888-4b30-a394-30edd3e366c4","Type":"ContainerStarted","Data":"07c9647a59a13f030609f4faa8377af1bf098e8509f073cf53ca02899420510b"}
Sep 18 19:43:25.246571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:25.246185    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:43:25.260602 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:25.260546    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2ghw8" podStartSLOduration=1.43603789 podStartE2EDuration="4.260522751s" totalImagesPullingTime="2.824484861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:25.260378978 +0000 UTC m=+313.778348620" watchObservedRunningTime="2026-09-18 19:43:25.260522751 +0000 UTC m=+313.778492406"
Sep 18 19:43:31.252591 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:43:31.252560    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2ghw8"
Sep 18 19:44:36.805494 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.805460    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cj8"]
Sep 18 19:44:36.809124 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.809100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:36.811888 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.811868    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 19:44:36.812013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.811917    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-szr5h\""
Sep 18 19:44:36.820002 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.819980    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-mhp2g"]
Sep 18 19:44:36.823450 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.823430    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cj8"]
Sep 18 19:44:36.823566 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.823535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.825909 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.825886    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:44:36.826060 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.826044    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:44:36.826142 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.826122    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ghg4f\""
Sep 18 19:44:36.832413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.832390    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mhp2g"]
Sep 18 19:44:36.895968 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.895940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjxd\" (UniqueName: \"kubernetes.io/projected/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-kube-api-access-qcjxd\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:36.896167 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.895978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.896167 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.896009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxxp\" (UniqueName: \"kubernetes.io/projected/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-kube-api-access-hrxxp\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.896167 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.896084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.896167 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.896138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:36.997234 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.997179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.997419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.997257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:36.997419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.997285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjxd\" (UniqueName: \"kubernetes.io/projected/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-kube-api-access-qcjxd\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:36.997419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.997312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.997419 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997347    2573 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:44:36.997419 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997391    2573 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 19:44:36.997604 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997426    2573 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 19:44:36.997604 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:36.997352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxxp\" (UniqueName: \"kubernetes.io/projected/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-kube-api-access-hrxxp\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:36.997604 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997430    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert podName:44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:37.497410305 +0000 UTC m=+386.015379937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert") pod "odh-model-controller-7b966d7984-mhp2g" (UID: "44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:44:36.997604 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997504    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs podName:95ba16ae-9ee4-4667-a900-83fa6c0ce16e nodeName:}" failed. No retries permitted until 2026-09-18 19:44:37.497486223 +0000 UTC m=+386.015455856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs") pod "model-serving-api-7b954ffc95-w9cj8" (UID: "95ba16ae-9ee4-4667-a900-83fa6c0ce16e") : secret "model-serving-api-tls" not found
Sep 18 19:44:36.997604 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:44:36.997521    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs podName:44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03 nodeName:}" failed. No retries permitted until 2026-09-18 19:44:37.49751272 +0000 UTC m=+386.015482341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs") pod "odh-model-controller-7b966d7984-mhp2g" (UID: "44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03") : secret "odh-model-controller-metrics-tls" not found
Sep 18 19:44:37.005616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.005586    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjxd\" (UniqueName: \"kubernetes.io/projected/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-kube-api-access-qcjxd\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:37.005616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.005607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxxp\" (UniqueName: \"kubernetes.io/projected/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-kube-api-access-hrxxp\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.501502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.501470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:37.501699 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.501527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.501699 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.501579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.504373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.504343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-cert\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.504497 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.504381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03-metrics-certs\") pod \"odh-model-controller-7b966d7984-mhp2g\" (UID: \"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03\") " pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.504497 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.504397    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ba16ae-9ee4-4667-a900-83fa6c0ce16e-tls-certs\") pod \"model-serving-api-7b954ffc95-w9cj8\" (UID: \"95ba16ae-9ee4-4667-a900-83fa6c0ce16e\") " pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:37.721336 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.721290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:37.733323 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.733273    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:37.996069 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:37.996034    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w9cj8"]
Sep 18 19:44:37.999019 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:44:37.998986    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ba16ae_9ee4_4667_a900_83fa6c0ce16e.slice/crio-20cb213aba15d00cadd4cd8ad71373aa3782b3024df24d6d174de0d175ab54f7 WatchSource:0}: Error finding container 20cb213aba15d00cadd4cd8ad71373aa3782b3024df24d6d174de0d175ab54f7: Status 404 returned error can't find the container with id 20cb213aba15d00cadd4cd8ad71373aa3782b3024df24d6d174de0d175ab54f7
Sep 18 19:44:38.011415 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:38.011387    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mhp2g"]
Sep 18 19:44:38.015004 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:44:38.014974    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e99a3c_f8fc_42dc_9bd8_2e8d5b3e6d03.slice/crio-01c1120ceac6eb43b2e1309f704872e4a18e73e1bea601cee356f8b4a23d9e86 WatchSource:0}: Error finding container 01c1120ceac6eb43b2e1309f704872e4a18e73e1bea601cee356f8b4a23d9e86: Status 404 returned error can't find the container with id 01c1120ceac6eb43b2e1309f704872e4a18e73e1bea601cee356f8b4a23d9e86
Sep 18 19:44:38.513181 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:38.513089    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mhp2g" event={"ID":"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03","Type":"ContainerStarted","Data":"01c1120ceac6eb43b2e1309f704872e4a18e73e1bea601cee356f8b4a23d9e86"}
Sep 18 19:44:38.515011 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:38.514949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w9cj8" event={"ID":"95ba16ae-9ee4-4667-a900-83fa6c0ce16e","Type":"ContainerStarted","Data":"20cb213aba15d00cadd4cd8ad71373aa3782b3024df24d6d174de0d175ab54f7"}
Sep 18 19:44:42.543793 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.543749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mhp2g" event={"ID":"44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03","Type":"ContainerStarted","Data":"44226aa3c1248df6291d32eac4693c53c33ba4815f4e998fa1cb4a81078b6986"}
Sep 18 19:44:42.544289 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.543893    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:42.545280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.545257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w9cj8" event={"ID":"95ba16ae-9ee4-4667-a900-83fa6c0ce16e","Type":"ContainerStarted","Data":"970bd7ae423bd6d1a1e1af829fd335e38652987e7388866995ec7d6e0d5a0dc7"}
Sep 18 19:44:42.545410 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.545395    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:42.561317 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.561268    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-mhp2g" podStartSLOduration=2.957079636 podStartE2EDuration="6.561250346s" totalImagesPullingTime="3.60417071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:42.559408026 +0000 UTC m=+391.077377665" watchObservedRunningTime="2026-09-18 19:44:42.561250346 +0000 UTC m=+391.079219986"
Sep 18 19:44:42.577998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:42.577956    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-w9cj8" podStartSLOduration=3.003642912 podStartE2EDuration="6.577941703s" totalImagesPullingTime="3.574298791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:42.576870149 +0000 UTC m=+391.094839790" watchObservedRunningTime="2026-09-18 19:44:42.577941703 +0000 UTC m=+391.095911342"
Sep 18 19:44:53.552021 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:53.551984    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-mhp2g"
Sep 18 19:44:53.553777 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:53.553755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-w9cj8"
Sep 18 19:44:54.320519 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.320489    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2krtg"]
Sep 18 19:44:54.323879 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.323855    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.326254 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.326234    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:44:54.331177 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.331153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2krtg"]
Sep 18 19:44:54.457549 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.457519    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sd6x\" (UniqueName: \"kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.457726 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.457563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.558818 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.558784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sd6x\" (UniqueName: \"kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.559222 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.558826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.559222 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.559186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.567313 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.567295    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sd6x\" (UniqueName: \"kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x\") pod \"s3-init-2krtg\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") " pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.652970 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.652938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2krtg"
Sep 18 19:44:54.921476 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:54.921383    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2krtg"]
Sep 18 19:44:54.924332 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:44:54.924299    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6a702e_2068_45e0_81e9_ecb8e4f6411c.slice/crio-03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7 WatchSource:0}: Error finding container 03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7: Status 404 returned error can't find the container with id 03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7
Sep 18 19:44:55.596554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:55.596522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerStarted","Data":"03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7"}
Sep 18 19:44:58.610541 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:44:58.610509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerStarted","Data":"d98067beb361baba11e47ad4b8324b1d17a251e5aca0cd1649e7a4e118d78f7d"}
Sep 18 19:45:00.619172 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:45:00.619137    2573 generic.go:356] "Generic (PLEG): container finished" podID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerID="d98067beb361baba11e47ad4b8324b1d17a251e5aca0cd1649e7a4e118d78f7d" exitCode=0
Sep 18 19:45:00.619172 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:45:00.619175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerDied","Data":"d98067beb361baba11e47ad4b8324b1d17a251e5aca0cd1649e7a4e118d78f7d"}
Sep 18 19:45:05.643421 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:45:05.643388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerStarted","Data":"a8a033ebea643506d38c3d43528ab5fa5f3e9d636efb442f31e8511d9fe26730"}
Sep 18 19:45:05.659696 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:45:05.659643    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2krtg" podStartSLOduration=1.882413256 podStartE2EDuration="11.659626204s" totalImagesPullingTime="7.996285166s" totalInitContainerRuntime="1.780927782s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:44:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:05.658684982 +0000 UTC m=+414.176654623" watchObservedRunningTime="2026-09-18 19:45:05.659626204 +0000 UTC m=+414.177595844"
Sep 18 19:46:10.899616 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:10.899581    2573 generic.go:356] "Generic (PLEG): container finished" podID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerID="a8a033ebea643506d38c3d43528ab5fa5f3e9d636efb442f31e8511d9fe26730" exitCode=0
Sep 18 19:46:10.900024 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:10.899653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerDied","Data":"a8a033ebea643506d38c3d43528ab5fa5f3e9d636efb442f31e8511d9fe26730"}
Sep 18 19:46:12.036289 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.036262    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2krtg"
Sep 18 19:46:12.126949 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.126913    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache\") pod \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") "
Sep 18 19:46:12.127135 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.126960    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sd6x\" (UniqueName: \"kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x\") pod \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\" (UID: \"ce6a702e-2068-45e0-81e9-ecb8e4f6411c\") "
Sep 18 19:46:12.129104 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.129072    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x" (OuterVolumeSpecName: "kube-api-access-4sd6x") pod "ce6a702e-2068-45e0-81e9-ecb8e4f6411c" (UID: "ce6a702e-2068-45e0-81e9-ecb8e4f6411c"). InnerVolumeSpecName "kube-api-access-4sd6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:12.145682 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.145652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce6a702e-2068-45e0-81e9-ecb8e4f6411c" (UID: "ce6a702e-2068-45e0-81e9-ecb8e4f6411c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:12.228451 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.228373    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-model-cache\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:12.228451 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.228402    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sd6x\" (UniqueName: \"kubernetes.io/projected/ce6a702e-2068-45e0-81e9-ecb8e4f6411c-kube-api-access-4sd6x\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:12.908672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.908645    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2krtg"
Sep 18 19:46:12.908672 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.908669    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2krtg" event={"ID":"ce6a702e-2068-45e0-81e9-ecb8e4f6411c","Type":"ContainerDied","Data":"03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7"}
Sep 18 19:46:12.908921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:12.908701    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f614307ddae9983b962e7d8d37c52ba8a3e3ddbee1d0c68467cb9f209573d7"
Sep 18 19:46:13.607155 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607124    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:13.607563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607467    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerName="download-hf-model"
Sep 18 19:46:13.607563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607478    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerName="download-hf-model"
Sep 18 19:46:13.607563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607488    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerName="s3-init"
Sep 18 19:46:13.607563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607493    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerName="s3-init"
Sep 18 19:46:13.607563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.607540    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6a702e-2068-45e0-81e9-ecb8e4f6411c" containerName="s3-init"
Sep 18 19:46:13.609443 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.609427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.611877 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.611854    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:46:13.617679 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.617657    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:13.640332 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.640306    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8jx\" (UniqueName: \"kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.640443 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.640354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.740878 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.740842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8jx\" (UniqueName: \"kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.741043 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.740897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.741286 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.741269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.750015 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.749992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8jx\" (UniqueName: \"kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx\") pod \"seaweedfs-tls-custom-6644bf5d6c-h4scs\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:13.918294 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:13.918166    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:14.186178 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:14.186132    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:14.188375 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:46:14.188345    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c91e6c_4382_411a_9ac4_0a61d4909b45.slice/crio-d7bd39d7fc430d185250d0d5f52145d6ce2691c0b82fdbf3ac092e15f0db3575 WatchSource:0}: Error finding container d7bd39d7fc430d185250d0d5f52145d6ce2691c0b82fdbf3ac092e15f0db3575: Status 404 returned error can't find the container with id d7bd39d7fc430d185250d0d5f52145d6ce2691c0b82fdbf3ac092e15f0db3575
Sep 18 19:46:14.456059 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:14.456037    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:46:14.917552 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:14.917508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" event={"ID":"60c91e6c-4382-411a-9ac4-0a61d4909b45","Type":"ContainerStarted","Data":"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"}
Sep 18 19:46:14.917960 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:14.917601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" event={"ID":"60c91e6c-4382-411a-9ac4-0a61d4909b45","Type":"ContainerStarted","Data":"d7bd39d7fc430d185250d0d5f52145d6ce2691c0b82fdbf3ac092e15f0db3575"}
Sep 18 19:46:14.933548 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:14.933494    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" podStartSLOduration=1.669799749 podStartE2EDuration="1.933477562s" totalImagesPullingTime="263.677813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:14.931845377 +0000 UTC m=+483.449815056" watchObservedRunningTime="2026-09-18 19:46:14.933477562 +0000 UTC m=+483.451447202"
Sep 18 19:46:15.615914 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:15.615875    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:16.924583 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:16.924540    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" podUID="60c91e6c-4382-411a-9ac4-0a61d4909b45" containerName="seaweedfs-tls-custom" containerID="cri-o://4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922" gracePeriod=30
Sep 18 19:46:18.150364 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.150340    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:18.280404 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.280315    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data\") pod \"60c91e6c-4382-411a-9ac4-0a61d4909b45\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") "
Sep 18 19:46:18.280404 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.280367    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl8jx\" (UniqueName: \"kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx\") pod \"60c91e6c-4382-411a-9ac4-0a61d4909b45\" (UID: \"60c91e6c-4382-411a-9ac4-0a61d4909b45\") "
Sep 18 19:46:18.281517 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.281495    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data" (OuterVolumeSpecName: "data") pod "60c91e6c-4382-411a-9ac4-0a61d4909b45" (UID: "60c91e6c-4382-411a-9ac4-0a61d4909b45"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:18.282571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.282546    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx" (OuterVolumeSpecName: "kube-api-access-vl8jx") pod "60c91e6c-4382-411a-9ac4-0a61d4909b45" (UID: "60c91e6c-4382-411a-9ac4-0a61d4909b45"). InnerVolumeSpecName "kube-api-access-vl8jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:18.381107 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.381071    2573 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/60c91e6c-4382-411a-9ac4-0a61d4909b45-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:18.381107 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.381100    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8jx\" (UniqueName: \"kubernetes.io/projected/60c91e6c-4382-411a-9ac4-0a61d4909b45-kube-api-access-vl8jx\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:18.933536 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.933502    2573 generic.go:356] "Generic (PLEG): container finished" podID="60c91e6c-4382-411a-9ac4-0a61d4909b45" containerID="4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922" exitCode=0
Sep 18 19:46:18.933741 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.933546    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"
Sep 18 19:46:18.933741 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.933598    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" event={"ID":"60c91e6c-4382-411a-9ac4-0a61d4909b45","Type":"ContainerDied","Data":"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"}
Sep 18 19:46:18.933741 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.933653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs" event={"ID":"60c91e6c-4382-411a-9ac4-0a61d4909b45","Type":"ContainerDied","Data":"d7bd39d7fc430d185250d0d5f52145d6ce2691c0b82fdbf3ac092e15f0db3575"}
Sep 18 19:46:18.933741 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.933679    2573 scope.go:117] "RemoveContainer" containerID="4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"
Sep 18 19:46:18.942941 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.942922    2573 scope.go:117] "RemoveContainer" containerID="4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"
Sep 18 19:46:18.943227 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:46:18.943189    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922\": container with ID starting with 4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922 not found: ID does not exist" containerID="4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"
Sep 18 19:46:18.943297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.943239    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922"} err="failed to get container status \"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922\": rpc error: code = NotFound desc = could not find container \"4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922\": container with ID starting with 4f45f897f21c55d93554f873cfcab3991ed4b81e08746e3181cb22befeb11922 not found: ID does not exist"
Sep 18 19:46:18.952805 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.952782    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:18.956780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:18.956760    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-h4scs"]
Sep 18 19:46:19.782247 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:19.782220    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c91e6c-4382-411a-9ac4-0a61d4909b45" path="/var/lib/kubelet/pods/60c91e6c-4382-411a-9ac4-0a61d4909b45/volumes"
Sep 18 19:46:23.495073 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.495041    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-vbnhs"]
Sep 18 19:46:23.495475 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.495400    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c91e6c-4382-411a-9ac4-0a61d4909b45" containerName="seaweedfs-tls-custom"
Sep 18 19:46:23.495475 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.495415    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c91e6c-4382-411a-9ac4-0a61d4909b45" containerName="seaweedfs-tls-custom"
Sep 18 19:46:23.495475 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.495465    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c91e6c-4382-411a-9ac4-0a61d4909b45" containerName="seaweedfs-tls-custom"
Sep 18 19:46:23.498549 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.498528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.500823 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.500801    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:46:23.500909 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.500879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:46:23.504891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.504863    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vbnhs"]
Sep 18 19:46:23.625374 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.625342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.625570 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.625391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vxh\" (UniqueName: \"kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.726045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.726014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.726255 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.726055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vxh\" (UniqueName: \"kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.726428 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.726408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.734085 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.734063    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vxh\" (UniqueName: \"kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh\") pod \"s3-tls-init-custom-vbnhs\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") " pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:23.828469 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:23.828392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:24.100503 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:24.100480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vbnhs"]
Sep 18 19:46:24.103076 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:46:24.103049    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73acbcc9_eb22_4f8f_a638_6bab293eba24.slice/crio-40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3 WatchSource:0}: Error finding container 40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3: Status 404 returned error can't find the container with id 40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3
Sep 18 19:46:24.959492 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:24.959454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerStarted","Data":"b47f14e3ae5c1bb9d82c3dcf5ea011de051cafce6089cafc5b3d3955c05d58c4"}
Sep 18 19:46:24.959912 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:24.959500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerStarted","Data":"40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3"}
Sep 18 19:46:25.964470 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:25.964432    2573 generic.go:356] "Generic (PLEG): container finished" podID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerID="b47f14e3ae5c1bb9d82c3dcf5ea011de051cafce6089cafc5b3d3955c05d58c4" exitCode=0
Sep 18 19:46:25.964926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:25.964489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerDied","Data":"b47f14e3ae5c1bb9d82c3dcf5ea011de051cafce6089cafc5b3d3955c05d58c4"}
Sep 18 19:46:26.969809 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:26.969772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerStarted","Data":"3109a87bcee31346b0b840051ee3618a12047e7c19e6c3e53f11ea1a2ad8cdb7"}
Sep 18 19:46:26.985069 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:26.985017    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-vbnhs" podStartSLOduration=2.231472564 podStartE2EDuration="3.984998664s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.7535261s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:26.984333239 +0000 UTC m=+495.502302892" watchObservedRunningTime="2026-09-18 19:46:26.984998664 +0000 UTC m=+495.502968305"
Sep 18 19:46:32.996779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:32.996742    2573 generic.go:356] "Generic (PLEG): container finished" podID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerID="3109a87bcee31346b0b840051ee3618a12047e7c19e6c3e53f11ea1a2ad8cdb7" exitCode=0
Sep 18 19:46:32.997132 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:32.996817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerDied","Data":"3109a87bcee31346b0b840051ee3618a12047e7c19e6c3e53f11ea1a2ad8cdb7"}
Sep 18 19:46:34.137436 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.137408    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:34.221528 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.221497    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5vxh\" (UniqueName: \"kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh\") pod \"73acbcc9-eb22-4f8f-a638-6bab293eba24\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") "
Sep 18 19:46:34.221701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.221563    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache\") pod \"73acbcc9-eb22-4f8f-a638-6bab293eba24\" (UID: \"73acbcc9-eb22-4f8f-a638-6bab293eba24\") "
Sep 18 19:46:34.223643 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.223615    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh" (OuterVolumeSpecName: "kube-api-access-h5vxh") pod "73acbcc9-eb22-4f8f-a638-6bab293eba24" (UID: "73acbcc9-eb22-4f8f-a638-6bab293eba24"). InnerVolumeSpecName "kube-api-access-h5vxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:34.243664 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.243628    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache" (OuterVolumeSpecName: "model-cache") pod "73acbcc9-eb22-4f8f-a638-6bab293eba24" (UID: "73acbcc9-eb22-4f8f-a638-6bab293eba24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:34.322165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.322091    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vxh\" (UniqueName: \"kubernetes.io/projected/73acbcc9-eb22-4f8f-a638-6bab293eba24-kube-api-access-h5vxh\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:34.322165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:34.322116    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73acbcc9-eb22-4f8f-a638-6bab293eba24-model-cache\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:35.011756 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:35.011727    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vbnhs"
Sep 18 19:46:35.011943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:35.011729    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vbnhs" event={"ID":"73acbcc9-eb22-4f8f-a638-6bab293eba24","Type":"ContainerDied","Data":"40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3"}
Sep 18 19:46:35.011943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:35.011853    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40249300e3ccccc69d0ae804a39c3184d644b19f080d97a13fa8a3eaf1e189a3"
Sep 18 19:46:37.767382 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767351    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-6s2f4"]
Sep 18 19:46:37.767764 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767673    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerName="download-hf-model"
Sep 18 19:46:37.767764 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767685    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerName="download-hf-model"
Sep 18 19:46:37.767764 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767692    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerName="s3-tls-init-custom"
Sep 18 19:46:37.767764 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767699    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerName="s3-tls-init-custom"
Sep 18 19:46:37.767764 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.767759    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="73acbcc9-eb22-4f8f-a638-6bab293eba24" containerName="s3-tls-init-custom"
Sep 18 19:46:37.770042 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.770023    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.772092 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.772063    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 19:46:37.772092 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.772086    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:46:37.783259 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.783236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6s2f4"]
Sep 18 19:46:37.849393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.849360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nhh\" (UniqueName: \"kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.849554 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.849435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.950691 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.950656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.950846 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.950819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nhh\" (UniqueName: \"kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.951087 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.951069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:37.958870 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:37.958844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nhh\" (UniqueName: \"kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh\") pod \"s3-tls-init-serving-6s2f4\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") " pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:38.088614 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:38.088520    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:38.355598 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:38.355563    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6s2f4"]
Sep 18 19:46:38.357845 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:46:38.357804    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47486c0a_e2e9_4b68_b23c_04bffe84b6fa.slice/crio-fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d WatchSource:0}: Error finding container fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d: Status 404 returned error can't find the container with id fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d
Sep 18 19:46:39.030109 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:39.030067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerStarted","Data":"597a4cba2476a5659a9301c6c58fdc44b0d3bc8fc7c62314662f73f17a314f46"}
Sep 18 19:46:39.030533 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:39.030117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerStarted","Data":"fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d"}
Sep 18 19:46:41.041461 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:41.041428    2573 generic.go:356] "Generic (PLEG): container finished" podID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerID="597a4cba2476a5659a9301c6c58fdc44b0d3bc8fc7c62314662f73f17a314f46" exitCode=0
Sep 18 19:46:41.041856 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:41.041498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerDied","Data":"597a4cba2476a5659a9301c6c58fdc44b0d3bc8fc7c62314662f73f17a314f46"}
Sep 18 19:46:42.047401 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:42.047369    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerStarted","Data":"45ebab59c5699b14b5c5d7dd0a7f7678ef3675a3192110094937c86814cb08fa"}
Sep 18 19:46:42.063000 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:42.062954    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-6s2f4" podStartSLOduration=3.183220489 podStartE2EDuration="5.062938209s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.87971772s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:42.062328959 +0000 UTC m=+510.580298598" watchObservedRunningTime="2026-09-18 19:46:42.062938209 +0000 UTC m=+510.580907850"
Sep 18 19:46:47.072907 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:47.072873    2573 generic.go:356] "Generic (PLEG): container finished" podID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerID="45ebab59c5699b14b5c5d7dd0a7f7678ef3675a3192110094937c86814cb08fa" exitCode=0
Sep 18 19:46:47.073279 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:47.072946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerDied","Data":"45ebab59c5699b14b5c5d7dd0a7f7678ef3675a3192110094937c86814cb08fa"}
Sep 18 19:46:48.209818 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.209791    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:48.240128 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.240103    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2nhh\" (UniqueName: \"kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh\") pod \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") "
Sep 18 19:46:48.240301 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.240160    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache\") pod \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\" (UID: \"47486c0a-e2e9-4b68-b23c-04bffe84b6fa\") "
Sep 18 19:46:48.242449 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.242414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh" (OuterVolumeSpecName: "kube-api-access-q2nhh") pod "47486c0a-e2e9-4b68-b23c-04bffe84b6fa" (UID: "47486c0a-e2e9-4b68-b23c-04bffe84b6fa"). InnerVolumeSpecName "kube-api-access-q2nhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:48.262784 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.262756    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "47486c0a-e2e9-4b68-b23c-04bffe84b6fa" (UID: "47486c0a-e2e9-4b68-b23c-04bffe84b6fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:48.341018 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.340951    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nhh\" (UniqueName: \"kubernetes.io/projected/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-kube-api-access-q2nhh\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:48.341018 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:48.340977    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47486c0a-e2e9-4b68-b23c-04bffe84b6fa-model-cache\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:46:49.082384 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:49.082359    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6s2f4"
Sep 18 19:46:49.082553 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:49.082359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6s2f4" event={"ID":"47486c0a-e2e9-4b68-b23c-04bffe84b6fa","Type":"ContainerDied","Data":"fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d"}
Sep 18 19:46:49.082553 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:46:49.082480    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa00e9ec9d68fd3b86d10877bab112b5ad6888c41fc5b96a536b4794cfb9528d"
Sep 18 19:47:00.503682 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.503602    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:47:00.504150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.503961    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerName="s3-tls-init-serving"
Sep 18 19:47:00.504150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.503974    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerName="s3-tls-init-serving"
Sep 18 19:47:00.504150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.504001    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerName="download-hf-model"
Sep 18 19:47:00.504150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.504009    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerName="download-hf-model"
Sep 18 19:47:00.504150 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.504078    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="47486c0a-e2e9-4b68-b23c-04bffe84b6fa" containerName="s3-tls-init-serving"
Sep 18 19:47:00.507454 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.507433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.509977 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.509951    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:47:00.510101 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.509956    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:47:00.510177 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.510149    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 18 19:47:00.510258 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.510157    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:47:00.510258 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.510228    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 19:47:00.510258 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.510248    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:47:00.510399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.510303    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 18 19:47:00.518113 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.518094    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:47:00.529548 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.529517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.529671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.529569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.529671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.529615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtmw\" (UniqueName: \"kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.529780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.529701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.529780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.529745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.630511 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.630691 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.630691 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.630813 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630715    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.630813 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtmw\" (UniqueName: \"kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.631015 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.630988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.631383 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.631359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.631467 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.631412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.633072 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.633054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.638923 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.638872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtmw\" (UniqueName: \"kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-blj8p\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.659241 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.659195    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:00.662893 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.662876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.665582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.665558    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c0d673-kube-rbac-proxy-sar-config\""
Sep 18 19:47:00.665582 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.665577    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-c0d673-dockercfg-4fsxj\""
Sep 18 19:47:00.665766 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.665607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-c0d673\""
Sep 18 19:47:00.665766 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.665704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c0d673-predictor-serving-cert\""
Sep 18 19:47:00.672470 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.672450    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:00.731888 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.731848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.732050 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.731904    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.732050 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.731952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.732050 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.731979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.732167 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.732076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9jp\" (UniqueName: \"kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.818052 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.817972    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:00.833120 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9jp\" (UniqueName: \"kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833390 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:00.833278    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-serving-cert: secret "isvc-init-fail-c0d673-predictor-serving-cert" not found
Sep 18 19:47:00.833390 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833390 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833390 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:00.833367    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls podName:d2c33d70-4701-4261-84ad-90518a0f83f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:47:01.333342838 +0000 UTC m=+529.851312462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls") pod "isvc-init-fail-c0d673-predictor-6df449b465-6snbz" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6") : secret "isvc-init-fail-c0d673-predictor-serving-cert" not found
Sep 18 19:47:00.833717 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.833934 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.833915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.834143 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.834122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:00.842162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:00.842131    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9jp\" (UniqueName: \"kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:01.099266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:01.096542    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:47:01.128912 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:01.128877    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerStarted","Data":"9ef209c16fd252ad6305757705fd1efd5701db1dc6ee5355fe9f2443e557ba4a"}
Sep 18 19:47:01.338985 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:01.338947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:01.339151 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:01.339094    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-serving-cert: secret "isvc-init-fail-c0d673-predictor-serving-cert" not found
Sep 18 19:47:01.339193 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:01.339155    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls podName:d2c33d70-4701-4261-84ad-90518a0f83f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:47:02.339141148 +0000 UTC m=+530.857110766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls") pod "isvc-init-fail-c0d673-predictor-6df449b465-6snbz" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6") : secret "isvc-init-fail-c0d673-predictor-serving-cert" not found
Sep 18 19:47:02.135953 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:02.135916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerStarted","Data":"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"}
Sep 18 19:47:02.348313 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:02.348277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:02.350786 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:02.350760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"isvc-init-fail-c0d673-predictor-6df449b465-6snbz\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") " pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:02.473738 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:02.473644    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:02.746686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:02.746659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:02.748546 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:47:02.748512    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c33d70_4701_4261_84ad_90518a0f83f6.slice/crio-3cdbe4d6784ab5a8f533d4198ff0181bb91edd2492a296435d4eec59936a4678 WatchSource:0}: Error finding container 3cdbe4d6784ab5a8f533d4198ff0181bb91edd2492a296435d4eec59936a4678: Status 404 returned error can't find the container with id 3cdbe4d6784ab5a8f533d4198ff0181bb91edd2492a296435d4eec59936a4678
Sep 18 19:47:03.142091 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:03.142049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerStarted","Data":"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"}
Sep 18 19:47:03.142662 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:03.142126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerStarted","Data":"3cdbe4d6784ab5a8f533d4198ff0181bb91edd2492a296435d4eec59936a4678"}
Sep 18 19:47:05.152595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:05.152556    2573 generic.go:356] "Generic (PLEG): container finished" podID="07e21935-436f-40c8-bb19-ad676fd448ed" containerID="a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3" exitCode=0
Sep 18 19:47:05.152958 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:05.152615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerDied","Data":"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"}
Sep 18 19:47:11.196412 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:11.196384    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/0.log"
Sep 18 19:47:11.196868 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:11.196424    2573 generic.go:356] "Generic (PLEG): container finished" podID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerID="03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84" exitCode=1
Sep 18 19:47:11.196868 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:11.196513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerDied","Data":"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"}
Sep 18 19:47:15.365017 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:15.364979    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:19.238694 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:19.238660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerStarted","Data":"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"}
Sep 18 19:47:19.240775 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:19.240754    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/0.log"
Sep 18 19:47:19.240890 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:19.240870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerStarted","Data":"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"}
Sep 18 19:47:19.240963 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:19.240942    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer" containerID="cri-o://ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97" gracePeriod=30
Sep 18 19:47:20.599375 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.599339    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:47:20.608294 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.608264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.611317 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.611133    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 18 19:47:20.611317 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.611143    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 19:47:20.612630 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.612605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:47:20.724416 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.724363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.724585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.724425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.724585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.724471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.724585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.724552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8wz\" (UniqueName: \"kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.724719 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.724614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.825360 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.825323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8wz\" (UniqueName: \"kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.825557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.825387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.825557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.825438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.825557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.825468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.825557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.825498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.826247 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.826193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.826394 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.826247    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.826909 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.826880    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.828790 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.828747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.834153 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.834132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8wz\" (UniqueName: \"kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-6f5mk\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:20.923454 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:20.923371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:21.723307 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:21.723289    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:47:21.726016 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:47:21.725984    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb92b0d_6e18_4f15_8245_290111987c25.slice/crio-f5f13aed3b32c162f48439603129dc7f3260b8795671e4e26bbd3fa5ac648ce4 WatchSource:0}: Error finding container f5f13aed3b32c162f48439603129dc7f3260b8795671e4e26bbd3fa5ac648ce4: Status 404 returned error can't find the container with id f5f13aed3b32c162f48439603129dc7f3260b8795671e4e26bbd3fa5ac648ce4
Sep 18 19:47:22.261241 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.261191    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerStarted","Data":"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"}
Sep 18 19:47:22.263062 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.263021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerStarted","Data":"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"}
Sep 18 19:47:22.263239 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.263086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerStarted","Data":"f5f13aed3b32c162f48439603129dc7f3260b8795671e4e26bbd3fa5ac648ce4"}
Sep 18 19:47:22.693975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.693945    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/1.log"
Sep 18 19:47:22.694493 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.694470    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/0.log"
Sep 18 19:47:22.699241 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.699222    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:22.846803 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.846773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr9jp\" (UniqueName: \"kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp\") pod \"d2c33d70-4701-4261-84ad-90518a0f83f6\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") "
Sep 18 19:47:22.847281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.846857    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location\") pod \"d2c33d70-4701-4261-84ad-90518a0f83f6\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") "
Sep 18 19:47:22.847281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.846878    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config\") pod \"d2c33d70-4701-4261-84ad-90518a0f83f6\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") "
Sep 18 19:47:22.847281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.846902    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") pod \"d2c33d70-4701-4261-84ad-90518a0f83f6\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") "
Sep 18 19:47:22.847281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.846949    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert\") pod \"d2c33d70-4701-4261-84ad-90518a0f83f6\" (UID: \"d2c33d70-4701-4261-84ad-90518a0f83f6\") "
Sep 18 19:47:22.847281 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.847176    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2c33d70-4701-4261-84ad-90518a0f83f6" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:47:22.847558 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.847340    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2c33d70-4701-4261-84ad-90518a0f83f6" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:22.847558 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.847412    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2c33d70-4701-4261-84ad-90518a0f83f6" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:47:22.849061 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.849038    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2c33d70-4701-4261-84ad-90518a0f83f6" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:47:22.849154 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.849139    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp" (OuterVolumeSpecName: "kube-api-access-rr9jp") pod "d2c33d70-4701-4261-84ad-90518a0f83f6" (UID: "d2c33d70-4701-4261-84ad-90518a0f83f6"). InnerVolumeSpecName "kube-api-access-rr9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:22.948489 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.948455    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c33d70-4701-4261-84ad-90518a0f83f6-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:47:22.948489 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.948487    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:47:22.948714 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.948501    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c33d70-4701-4261-84ad-90518a0f83f6-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:47:22.948714 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.948515    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c33d70-4701-4261-84ad-90518a0f83f6-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:47:22.948714 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:22.948528    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9jp\" (UniqueName: \"kubernetes.io/projected/d2c33d70-4701-4261-84ad-90518a0f83f6-kube-api-access-rr9jp\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:47:23.270233 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270158    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/1.log"
Sep 18 19:47:23.270567 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270551    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c0d673-predictor-6df449b465-6snbz_d2c33d70-4701-4261-84ad-90518a0f83f6/storage-initializer/0.log"
Sep 18 19:47:23.270624 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270587    2573 generic.go:356] "Generic (PLEG): container finished" podID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerID="ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97" exitCode=1
Sep 18 19:47:23.270688 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"
Sep 18 19:47:23.270771 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270690    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerDied","Data":"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"}
Sep 18 19:47:23.270870 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz" event={"ID":"d2c33d70-4701-4261-84ad-90518a0f83f6","Type":"ContainerDied","Data":"3cdbe4d6784ab5a8f533d4198ff0181bb91edd2492a296435d4eec59936a4678"}
Sep 18 19:47:23.270870 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.270845    2573 scope.go:117] "RemoveContainer" containerID="ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"
Sep 18 19:47:23.282141 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.281386    2573 scope.go:117] "RemoveContainer" containerID="03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"
Sep 18 19:47:23.292365 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.292345    2573 scope.go:117] "RemoveContainer" containerID="ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"
Sep 18 19:47:23.292682 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:23.292656    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97\": container with ID starting with ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97 not found: ID does not exist" containerID="ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"
Sep 18 19:47:23.292770 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.292688    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97"} err="failed to get container status \"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97\": rpc error: code = NotFound desc = could not find container \"ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97\": container with ID starting with ea4f32790da3388a745a67294d3d9853320fdf45eca4c06642e4ce336fb96b97 not found: ID does not exist"
Sep 18 19:47:23.292770 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.292705    2573 scope.go:117] "RemoveContainer" containerID="03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"
Sep 18 19:47:23.293012 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:47:23.292989    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84\": container with ID starting with 03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84 not found: ID does not exist" containerID="03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"
Sep 18 19:47:23.293079 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.293019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84"} err="failed to get container status \"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84\": rpc error: code = NotFound desc = could not find container \"03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84\": container with ID starting with 03d43cf6b66c24728aca9d72370d166f4b9ea0d104ff3f8aa22be7318af83f84 not found: ID does not exist"
Sep 18 19:47:23.313727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.313697    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:23.319184 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.319153    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c0d673-predictor-6df449b465-6snbz"]
Sep 18 19:47:23.784045 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:23.784004    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" path="/var/lib/kubelet/pods/d2c33d70-4701-4261-84ad-90518a0f83f6/volumes"
Sep 18 19:47:25.283025 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:25.282993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerStarted","Data":"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"}
Sep 18 19:47:25.283439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:25.283159    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:25.302247 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:25.302180    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podStartSLOduration=2.207450578 podStartE2EDuration="25.302160539s" totalImagesPullingTime="19.198802434s" totalInitContainerRuntime="3.895907527s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:25.301088131 +0000 UTC m=+553.819057771" watchObservedRunningTime="2026-09-18 19:47:25.302160539 +0000 UTC m=+553.820130176"
Sep 18 19:47:26.288938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.288903    2573 generic.go:356] "Generic (PLEG): container finished" podID="6cb92b0d-6e18-4f15-8245-290111987c25" containerID="19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b" exitCode=0
Sep 18 19:47:26.289377 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.288949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerDied","Data":"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"}
Sep 18 19:47:26.289377 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.289348    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:26.289491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.289414    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:26.291250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.291228    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:26.291943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.291905    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:47:26.294886 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:26.294868    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:47:27.298051 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:27.297648    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:27.298847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:27.298773    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:47:28.305399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:28.304817    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:28.305399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:28.305341    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:47:38.305120 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:38.305070    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:38.305686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:38.305505    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:47:48.305399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:48.305362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:48.305830 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:48.305807    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:47:49.423393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.423359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerStarted","Data":"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"}
Sep 18 19:47:49.423859 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.423441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerStarted","Data":"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"}
Sep 18 19:47:49.423859 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.423749    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:49.423938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.423905    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:49.425136 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.425113    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:47:49.443647 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:49.443604    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podStartSLOduration=3.596577174 podStartE2EDuration="29.443590609s" totalImagesPullingTime="22.015927051s" totalInitContainerRuntime="3.831086384s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:49.441782612 +0000 UTC m=+577.959752254" watchObservedRunningTime="2026-09-18 19:47:49.443590609 +0000 UTC m=+577.961560248"
Sep 18 19:47:50.428336 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:50.428288    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:47:51.433493 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:51.433454    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:47:56.438451 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:56.438422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:47:56.439067 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:56.439042    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:47:58.305102 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:58.305055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:47:58.305552 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:47:58.305517    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:48:06.439014 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:06.438975    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:08.304847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:08.304793    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:48:08.305275 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:08.305247    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:48:16.439360 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:16.439319    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:18.305006 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:18.304956    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:48:18.305468 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:18.305445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:48:26.439739 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:26.439652    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:28.305294 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:28.305241    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:48:28.305708 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:28.305684    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:48:30.777895 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:30.777849    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:48:30.778352 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:30.778244    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:48:36.439735 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:36.439699    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:40.779419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:40.779384    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:48:40.779866 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:40.779534    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:48:46.439148 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:46.439103    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:55.573887 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.573849    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:48:55.574384 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.574133    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" containerID="cri-o://fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f" gracePeriod=30
Sep 18 19:48:55.574384 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.574152    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" containerID="cri-o://e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596" gracePeriod=30
Sep 18 19:48:55.574384 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.574192    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" containerID="cri-o://eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86" gracePeriod=30
Sep 18 19:48:55.587030 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587007    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:48:55.587590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587553    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.587590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587590    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.587718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587611    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.587718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587618    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.587718 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587710    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.587828 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.587723    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c33d70-4701-4261-84ad-90518a0f83f6" containerName="storage-initializer"
Sep 18 19:48:55.590034 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.590019    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.592287 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.592265    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 18 19:48:55.592386 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.592290    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 18 19:48:55.602472 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.602450    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:48:55.663788 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.663750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzzw\" (UniqueName: \"kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.663788 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.663793    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.664026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.663809    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.664026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.663835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.664026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.663908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.702569 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.702542    2573 generic.go:356] "Generic (PLEG): container finished" podID="07e21935-436f-40c8-bb19-ad676fd448ed" containerID="eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86" exitCode=2
Sep 18 19:48:55.702700 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.702608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerDied","Data":"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"}
Sep 18 19:48:55.765017 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.764982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzzw\" (UniqueName: \"kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765517 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765497    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.765827 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.765810    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.767693 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.767672    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.772428 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.772406    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzzw\" (UniqueName: \"kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:55.901162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:55.901123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:48:56.177189 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:56.177116    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:48:56.290412 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:56.290362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:48:56.439295 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:56.439186    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:48:56.709399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:56.709313    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerStarted","Data":"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"}
Sep 18 19:48:56.709399 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:48:56.709392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerStarted","Data":"e7823b8866969bed60bc1a5d52a8e1d79de6604cbb0590d046562084b1afd518"}
Sep 18 19:49:00.728173 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:00.728140    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerID="c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa" exitCode=0
Sep 18 19:49:00.728671 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:00.728237    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerDied","Data":"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"}
Sep 18 19:49:00.778839 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:00.778494    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:49:00.778947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:00.778865    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:01.289919 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.289871    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:49:01.736351 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.736316    2573 generic.go:356] "Generic (PLEG): container finished" podID="07e21935-436f-40c8-bb19-ad676fd448ed" containerID="fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f" exitCode=0
Sep 18 19:49:01.736763 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.736385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerDied","Data":"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"}
Sep 18 19:49:01.738794 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.738773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerStarted","Data":"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"}
Sep 18 19:49:01.738913 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.738823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerStarted","Data":"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"}
Sep 18 19:49:01.738913 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.738851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerStarted","Data":"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"}
Sep 18 19:49:01.739077 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.739060    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:49:01.759624 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:01.759579    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podStartSLOduration=3.07387219 podStartE2EDuration="6.759565019s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685692829s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:01.758106434 +0000 UTC m=+650.276076101" watchObservedRunningTime="2026-09-18 19:49:01.759565019 +0000 UTC m=+650.277534659"
Sep 18 19:49:02.742816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:02.742776    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:49:02.743222 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:02.742892    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:49:02.744401 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:02.744344    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:02.744956 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:02.744923    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:03.747175 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:03.747126    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:03.747740 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:03.747514    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:06.289595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:06.289555    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:49:06.290025 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:06.289713    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:49:06.439787 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:06.439751    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:49:08.751337 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:08.751308    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:49:08.751985 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:08.751952    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:08.752105 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:08.752081    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:10.777940 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:10.777898    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:49:10.778383 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:10.778229    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:11.290254 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:11.290191    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:49:12.778363 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:12.778334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:49:16.290246 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:16.290186    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:49:18.751969 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:18.751918    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:18.752456 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:18.752433    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:20.777936 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:20.777891    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:49:20.778421 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:20.778064    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:49:20.778421 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:20.778343    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:20.778520 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:20.778477    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:49:21.040425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.040347    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:49:21.040649 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.040599    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" containerID="cri-o://05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03" gracePeriod=30
Sep 18 19:49:21.040797 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.040698    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy" containerID="cri-o://9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249" gracePeriod=30
Sep 18 19:49:21.290212 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.290171    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 18 19:49:21.434703 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.434648    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 19:49:21.833390 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.833301    2573 generic.go:356] "Generic (PLEG): container finished" podID="6cb92b0d-6e18-4f15-8245-290111987c25" containerID="9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249" exitCode=2
Sep 18 19:49:21.833390 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:21.833373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerDied","Data":"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"}
Sep 18 19:49:22.778060 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:22.778020    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 19:49:25.719238 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.719213    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:49:25.829651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.829572    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls\") pod \"07e21935-436f-40c8-bb19-ad676fd448ed\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") "
Sep 18 19:49:25.829651 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.829643    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jtmw\" (UniqueName: \"kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw\") pod \"07e21935-436f-40c8-bb19-ad676fd448ed\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") "
Sep 18 19:49:25.829877 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.829689    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config\") pod \"07e21935-436f-40c8-bb19-ad676fd448ed\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") "
Sep 18 19:49:25.829942 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.829889    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert\") pod \"07e21935-436f-40c8-bb19-ad676fd448ed\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") "
Sep 18 19:49:25.830012 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.829950    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location\") pod \"07e21935-436f-40c8-bb19-ad676fd448ed\" (UID: \"07e21935-436f-40c8-bb19-ad676fd448ed\") "
Sep 18 19:49:25.830179 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.830029    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07e21935-436f-40c8-bb19-ad676fd448ed" (UID: "07e21935-436f-40c8-bb19-ad676fd448ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:49:25.830341 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.830219    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07e21935-436f-40c8-bb19-ad676fd448ed" (UID: "07e21935-436f-40c8-bb19-ad676fd448ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:49:25.830341 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.830315    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:25.830477 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.830351    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07e21935-436f-40c8-bb19-ad676fd448ed" (UID: "07e21935-436f-40c8-bb19-ad676fd448ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:25.831841 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.831816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07e21935-436f-40c8-bb19-ad676fd448ed" (UID: "07e21935-436f-40c8-bb19-ad676fd448ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:49:25.832030 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.832003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw" (OuterVolumeSpecName: "kube-api-access-8jtmw") pod "07e21935-436f-40c8-bb19-ad676fd448ed" (UID: "07e21935-436f-40c8-bb19-ad676fd448ed"). InnerVolumeSpecName "kube-api-access-8jtmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:49:25.851813 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.851786    2573 generic.go:356] "Generic (PLEG): container finished" podID="07e21935-436f-40c8-bb19-ad676fd448ed" containerID="e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596" exitCode=0
Sep 18 19:49:25.851917 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.851855    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"
Sep 18 19:49:25.851994 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.851857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerDied","Data":"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"}
Sep 18 19:49:25.852052 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.852001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p" event={"ID":"07e21935-436f-40c8-bb19-ad676fd448ed","Type":"ContainerDied","Data":"9ef209c16fd252ad6305757705fd1efd5701db1dc6ee5355fe9f2443e557ba4a"}
Sep 18 19:49:25.852106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.852055    2573 scope.go:117] "RemoveContainer" containerID="e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"
Sep 18 19:49:25.860264 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.860248    2573 scope.go:117] "RemoveContainer" containerID="eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"
Sep 18 19:49:25.867813 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.867797    2573 scope.go:117] "RemoveContainer" containerID="fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"
Sep 18 19:49:25.872920 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.872897    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:49:25.876134 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.876119    2573 scope.go:117] "RemoveContainer" containerID="a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"
Sep 18 19:49:25.876525 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.876507    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-blj8p"]
Sep 18 19:49:25.883496 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.883478    2573 scope.go:117] "RemoveContainer" containerID="e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"
Sep 18 19:49:25.883743 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:25.883723    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596\": container with ID starting with e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596 not found: ID does not exist" containerID="e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"
Sep 18 19:49:25.883815 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.883749    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596"} err="failed to get container status \"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596\": rpc error: code = NotFound desc = could not find container \"e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596\": container with ID starting with e86b6b8327c00bcf3ac4af1f40b24588e02c7e336b7198f07f71d4053d1ab596 not found: ID does not exist"
Sep 18 19:49:25.883815 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.883768    2573 scope.go:117] "RemoveContainer" containerID="eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"
Sep 18 19:49:25.884047 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:25.884028    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86\": container with ID starting with eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86 not found: ID does not exist" containerID="eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"
Sep 18 19:49:25.884108 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.884048    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86"} err="failed to get container status \"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86\": rpc error: code = NotFound desc = could not find container \"eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86\": container with ID starting with eef7d6de0bc24104f631709a8cda1c7d1050935b6dee125813bbe65575db4b86 not found: ID does not exist"
Sep 18 19:49:25.884108 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.884062    2573 scope.go:117] "RemoveContainer" containerID="fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"
Sep 18 19:49:25.884311 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:25.884294    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f\": container with ID starting with fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f not found: ID does not exist" containerID="fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"
Sep 18 19:49:25.884360 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.884315    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f"} err="failed to get container status \"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f\": rpc error: code = NotFound desc = could not find container \"fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f\": container with ID starting with fe5d6a48c1517e189e9299c228090ad90e2281d3baaeb5f06c293079bc5e520f not found: ID does not exist"
Sep 18 19:49:25.884360 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.884335    2573 scope.go:117] "RemoveContainer" containerID="a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"
Sep 18 19:49:25.884550 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:25.884534    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3\": container with ID starting with a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3 not found: ID does not exist" containerID="a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"
Sep 18 19:49:25.884602 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.884558    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3"} err="failed to get container status \"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3\": rpc error: code = NotFound desc = could not find container \"a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3\": container with ID starting with a41590d026a020b77d9f20ed9e154bea7bb35b14e86d96a8776b91b2cc3c5ff3 not found: ID does not exist"
Sep 18 19:49:25.931308 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.931275    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07e21935-436f-40c8-bb19-ad676fd448ed-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:25.931308 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.931311    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07e21935-436f-40c8-bb19-ad676fd448ed-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:25.931498 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.931324    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07e21935-436f-40c8-bb19-ad676fd448ed-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:25.931498 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:25.931336    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtmw\" (UniqueName: \"kubernetes.io/projected/07e21935-436f-40c8-bb19-ad676fd448ed-kube-api-access-8jtmw\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.227728 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.227688    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:49:26.228043 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228030    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container"
Sep 18 19:49:26.228043 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228043    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228055    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="storage-initializer"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228062    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="storage-initializer"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228071    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228077    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228088    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228094    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228138    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kube-rbac-proxy"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228147    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="agent"
Sep 18 19:49:26.228165 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.228156    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" containerName="kserve-container"
Sep 18 19:49:26.231053 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.231031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.233347 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.233322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 18 19:49:26.233461 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.233420    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 19:49:26.240802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.240776    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:49:26.333701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.333666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbtq\" (UniqueName: \"kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.333927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.333757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.333927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.333815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.333927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.333862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.333927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.333918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434510 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbtq\" (UniqueName: \"kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434510 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434512    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434724 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434724 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434724 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.434724 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:26.434685    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 18 19:49:26.434936 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:26.434762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls podName:9a198c14-f918-43d3-98ff-3e586474f0ba nodeName:}" failed. No retries permitted until 2026-09-18 19:49:26.934741404 +0000 UTC m=+675.452711038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-whrdt" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 18 19:49:26.435003 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.434989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.435273 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.435251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.435386 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.435335    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.442784 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.442762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbtq\" (UniqueName: \"kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.477560 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.477540    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:49:26.534983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.534911    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx8wz\" (UniqueName: \"kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz\") pod \"6cb92b0d-6e18-4f15-8245-290111987c25\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") "
Sep 18 19:49:26.534983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.534943    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert\") pod \"6cb92b0d-6e18-4f15-8245-290111987c25\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") "
Sep 18 19:49:26.534983 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.534978    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config\") pod \"6cb92b0d-6e18-4f15-8245-290111987c25\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") "
Sep 18 19:49:26.535277 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.535026    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls\") pod \"6cb92b0d-6e18-4f15-8245-290111987c25\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") "
Sep 18 19:49:26.535277 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.535076    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location\") pod \"6cb92b0d-6e18-4f15-8245-290111987c25\" (UID: \"6cb92b0d-6e18-4f15-8245-290111987c25\") "
Sep 18 19:49:26.535455 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.535430    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6cb92b0d-6e18-4f15-8245-290111987c25" (UID: "6cb92b0d-6e18-4f15-8245-290111987c25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:49:26.535455 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.535440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6cb92b0d-6e18-4f15-8245-290111987c25" (UID: "6cb92b0d-6e18-4f15-8245-290111987c25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:49:26.535588 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.535428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cb92b0d-6e18-4f15-8245-290111987c25" (UID: "6cb92b0d-6e18-4f15-8245-290111987c25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:49:26.537031 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.537009    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz" (OuterVolumeSpecName: "kube-api-access-bx8wz") pod "6cb92b0d-6e18-4f15-8245-290111987c25" (UID: "6cb92b0d-6e18-4f15-8245-290111987c25"). InnerVolumeSpecName "kube-api-access-bx8wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:49:26.537107 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.537064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6cb92b0d-6e18-4f15-8245-290111987c25" (UID: "6cb92b0d-6e18-4f15-8245-290111987c25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:49:26.636374 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.636341    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8wz\" (UniqueName: \"kubernetes.io/projected/6cb92b0d-6e18-4f15-8245-290111987c25-kube-api-access-bx8wz\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.636374 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.636369    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.636526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.636383    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cb92b0d-6e18-4f15-8245-290111987c25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.636526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.636396    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cb92b0d-6e18-4f15-8245-290111987c25-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.636526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.636409    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cb92b0d-6e18-4f15-8245-290111987c25-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:49:26.859430 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.859400    2573 generic.go:356] "Generic (PLEG): container finished" podID="6cb92b0d-6e18-4f15-8245-290111987c25" containerID="05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03" exitCode=0
Sep 18 19:49:26.859816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.859436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerDied","Data":"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"}
Sep 18 19:49:26.859816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.859458    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"
Sep 18 19:49:26.859816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.859485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" event={"ID":"6cb92b0d-6e18-4f15-8245-290111987c25","Type":"ContainerDied","Data":"f5f13aed3b32c162f48439603129dc7f3260b8795671e4e26bbd3fa5ac648ce4"}
Sep 18 19:49:26.859816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.859511    2573 scope.go:117] "RemoveContainer" containerID="9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"
Sep 18 19:49:26.867561 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.867540    2573 scope.go:117] "RemoveContainer" containerID="05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"
Sep 18 19:49:26.875780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.875761    2573 scope.go:117] "RemoveContainer" containerID="19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"
Sep 18 19:49:26.881017 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.880996    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:49:26.883361 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.883345    2573 scope.go:117] "RemoveContainer" containerID="9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"
Sep 18 19:49:26.883681 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:26.883659    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249\": container with ID starting with 9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249 not found: ID does not exist" containerID="9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"
Sep 18 19:49:26.883779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.883693    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249"} err="failed to get container status \"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249\": rpc error: code = NotFound desc = could not find container \"9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249\": container with ID starting with 9c189dc5ec0ce0762ee9c7885cd2aa0eccb24951660963ccd7632ebacd519249 not found: ID does not exist"
Sep 18 19:49:26.883779 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.883711    2573 scope.go:117] "RemoveContainer" containerID="05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"
Sep 18 19:49:26.883957 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:26.883937    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03\": container with ID starting with 05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03 not found: ID does not exist" containerID="05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"
Sep 18 19:49:26.883997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.883969    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03"} err="failed to get container status \"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03\": rpc error: code = NotFound desc = could not find container \"05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03\": container with ID starting with 05f59519db01bcc799ba2c38c4b27280d446fa00fc3bbc11a5486e1b390daa03 not found: ID does not exist"
Sep 18 19:49:26.883997 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.883985    2573 scope.go:117] "RemoveContainer" containerID="19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"
Sep 18 19:49:26.884247 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:49:26.884225    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b\": container with ID starting with 19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b not found: ID does not exist" containerID="19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"
Sep 18 19:49:26.884301 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.884256    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b"} err="failed to get container status \"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b\": rpc error: code = NotFound desc = could not find container \"19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b\": container with ID starting with 19adf4f122a44204cf45b6a4678e69f52885cdcca6c23b420e4544bd8342fb1b not found: ID does not exist"
Sep 18 19:49:26.887122 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.887102    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk"]
Sep 18 19:49:26.939007 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.938980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:26.941340 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:26.941317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-whrdt\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:27.142444 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.142352    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:27.413686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.413604    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:49:27.416702 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:49:27.416669    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a198c14_f918_43d3_98ff_3e586474f0ba.slice/crio-266e179801c909f4f6d63ccaa7e907e8796eae9d8bfa49477f24278cff3efda1 WatchSource:0}: Error finding container 266e179801c909f4f6d63ccaa7e907e8796eae9d8bfa49477f24278cff3efda1: Status 404 returned error can't find the container with id 266e179801c909f4f6d63ccaa7e907e8796eae9d8bfa49477f24278cff3efda1
Sep 18 19:49:27.435296 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.435255    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-6f5mk" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": context deadline exceeded"
Sep 18 19:49:27.783776 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.783696    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e21935-436f-40c8-bb19-ad676fd448ed" path="/var/lib/kubelet/pods/07e21935-436f-40c8-bb19-ad676fd448ed/volumes"
Sep 18 19:49:27.784548 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.784528    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" path="/var/lib/kubelet/pods/6cb92b0d-6e18-4f15-8245-290111987c25/volumes"
Sep 18 19:49:27.866840 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.866804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerStarted","Data":"87b5c152f126a2fec1e2bdf19180c8cf848c6621e7870d3d7a3bafe87fdb0814"}
Sep 18 19:49:27.867247 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:27.866862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerStarted","Data":"266e179801c909f4f6d63ccaa7e907e8796eae9d8bfa49477f24278cff3efda1"}
Sep 18 19:49:28.751851 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:28.751808    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:28.752228 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:28.752185    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:31.885994 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:31.885961    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerID="87b5c152f126a2fec1e2bdf19180c8cf848c6621e7870d3d7a3bafe87fdb0814" exitCode=0
Sep 18 19:49:31.886422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:31.886031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerDied","Data":"87b5c152f126a2fec1e2bdf19180c8cf848c6621e7870d3d7a3bafe87fdb0814"}
Sep 18 19:49:32.893706 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:32.893671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerStarted","Data":"98c9e143f65967f735313ce61d3a198774cf9bb197b73049e488f790b13995bc"}
Sep 18 19:49:32.894131 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:32.893734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerStarted","Data":"d7a8347708d89aa12938022788f46c11bc581af07258f75e9e8a9c2a0f326a9f"}
Sep 18 19:49:32.894131 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:32.893934    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:32.912897 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:32.912850    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podStartSLOduration=3.162961758 podStartE2EDuration="6.912837721s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.749875963s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:32.911600534 +0000 UTC m=+681.429570201" watchObservedRunningTime="2026-09-18 19:49:32.912837721 +0000 UTC m=+681.430807360"
Sep 18 19:49:33.898464 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:33.898434    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:33.899541 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:33.899517    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:49:34.902556 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:34.902516    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:49:38.752439 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:38.752400    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:38.752891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:38.752766    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:39.907382 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:39.907353    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:49:39.907831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:39.907805    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:49:48.752406 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:48.752361    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:48.752894 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:48.752810    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:49.908265 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:49.908225    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:49:58.752590 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:58.752483    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:49:58.753109 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:58.752916    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:49:59.908111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:49:59.908068    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:50:08.751994 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:08.751950    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:50:08.752505 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:08.752439    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:09.908146 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:09.908106    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:50:18.752455 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:18.752426    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:50:18.752894 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:18.752874    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:50:19.912048 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:19.912003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:50:29.908129 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:29.908089    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:50:30.608299 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.608267    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:50:30.608594 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.608549    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" containerID="cri-o://c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d" gracePeriod=30
Sep 18 19:50:30.608703 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.608612    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" containerID="cri-o://f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1" gracePeriod=30
Sep 18 19:50:30.608703 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.608567    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" containerID="cri-o://0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664" gracePeriod=30
Sep 18 19:50:30.704502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704470    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:50:30.704889 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704873    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="storage-initializer"
Sep 18 19:50:30.704975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704892    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="storage-initializer"
Sep 18 19:50:30.704975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704903    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container"
Sep 18 19:50:30.704975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704910    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container"
Sep 18 19:50:30.704975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704932    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy"
Sep 18 19:50:30.704975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.704941    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy"
Sep 18 19:50:30.705318 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.705024    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kserve-container"
Sep 18 19:50:30.705318 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.705041    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb92b0d-6e18-4f15-8245-290111987c25" containerName="kube-rbac-proxy"
Sep 18 19:50:30.708340 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.708319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.710441 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.710418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 18 19:50:30.710542 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.710458    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 18 19:50:30.717888 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.717866    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:50:30.892500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.892399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.892500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.892447    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.892500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.892483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7plxf\" (UniqueName: \"kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.993865 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.993823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.993865 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.993869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.994415 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.993992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7plxf\" (UniqueName: \"kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:30.994415 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:50:30.994004    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 18 19:50:30.994415 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:50:30.994081    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls podName:abf6a7b2-2ead-49ba-ac48-850a8e95b3ab nodeName:}" failed. No retries permitted until 2026-09-18 19:50:31.494058282 +0000 UTC m=+740.012027905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-65ngk" (UID: "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab") : secret "message-dumper-predictor-serving-cert" not found
Sep 18 19:50:30.994710 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:30.994693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:31.004459 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.004425    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7plxf\" (UniqueName: \"kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:31.143709 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.143631    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerID="f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1" exitCode=2
Sep 18 19:50:31.143853 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.143700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerDied","Data":"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"}
Sep 18 19:50:31.498743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.498634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:31.501266 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.501235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-65ngk\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:31.620372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.620332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:31.891775 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.891745    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:50:31.893542 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:50:31.893512    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf6a7b2_2ead_49ba_ac48_850a8e95b3ab.slice/crio-94d22ff85f793c7822402959f0db9b1e8b7cf4fe974d9038ad9e283f80a71003 WatchSource:0}: Error finding container 94d22ff85f793c7822402959f0db9b1e8b7cf4fe974d9038ad9e283f80a71003: Status 404 returned error can't find the container with id 94d22ff85f793c7822402959f0db9b1e8b7cf4fe974d9038ad9e283f80a71003
Sep 18 19:50:31.895655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:31.895636    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:50:32.149300 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:32.149196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerStarted","Data":"94d22ff85f793c7822402959f0db9b1e8b7cf4fe974d9038ad9e283f80a71003"}
Sep 18 19:50:33.748100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:33.748056    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:34.159814 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:34.159774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerStarted","Data":"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"}
Sep 18 19:50:34.159988 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:34.159847    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerStarted","Data":"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"}
Sep 18 19:50:34.160046 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:34.160031    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:34.178798 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:34.178748    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" podStartSLOduration=2.895359867 podStartE2EDuration="4.178733259s" totalImagesPullingTime="1.283373392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:34.177009588 +0000 UTC m=+742.694979229" watchObservedRunningTime="2026-09-18 19:50:34.178733259 +0000 UTC m=+742.696702896"
Sep 18 19:50:35.164890 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:35.164855    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:35.166452 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:35.166429    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:36.172192 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:36.172153    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerID="c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d" exitCode=0
Sep 18 19:50:36.172690 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:36.172230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerDied","Data":"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"}
Sep 18 19:50:38.747452 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:38.747412    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:38.752240 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:38.752188    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:38.753016 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:38.752994    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:50:39.908473 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:39.908432    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:50:42.187250 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:42.187192    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:50:43.747542 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:43.747502    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:43.747947 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:43.747641    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:50:48.747824 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:48.747779    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:48.752615 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:48.752583    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:48.753726 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:48.753704    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:50:49.908412 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:49.908382    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:50:50.747481 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.747446    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:50:50.750767 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.750746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.752938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.752921    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 18 19:50:50.753036 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.752951    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 18 19:50:50.761342 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.761320    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:50:50.862397 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.862354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.862593 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.862463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmqr\" (UniqueName: \"kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.862593 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.862484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.862593 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.862547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.862593 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.862585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.962979 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.962948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmqr\" (UniqueName: \"kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.962979 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.962986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963183    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963538 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:50:50.963437    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 18 19:50:50.963704 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:50:50.963558    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls podName:bfbc3440-4645-4483-bf41-67c7ee10f761 nodeName:}" failed. No retries permitted until 2026-09-18 19:50:51.463536899 +0000 UTC m=+759.981506524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls") pod "isvc-logger-predictor-85d5cb84f6-57qgh" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761") : secret "isvc-logger-predictor-serving-cert" not found
Sep 18 19:50:50.963704 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963560    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.963826 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.963808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:50.973574 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:50.973551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmqr\" (UniqueName: \"kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:51.466958 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:51.466917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:51.469523 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:51.469494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-57qgh\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:51.661890 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:51.661843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:51.932778 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:51.932749    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:50:51.934706 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:50:51.934681    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbc3440_4645_4483_bf41_67c7ee10f761.slice/crio-58ee24c59317aaf184514bec67a85f99f0d5b455f0babaf15e9ad8100ade72d2 WatchSource:0}: Error finding container 58ee24c59317aaf184514bec67a85f99f0d5b455f0babaf15e9ad8100ade72d2: Status 404 returned error can't find the container with id 58ee24c59317aaf184514bec67a85f99f0d5b455f0babaf15e9ad8100ade72d2
Sep 18 19:50:52.244342 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:52.244251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerStarted","Data":"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"}
Sep 18 19:50:52.244342 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:52.244315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerStarted","Data":"58ee24c59317aaf184514bec67a85f99f0d5b455f0babaf15e9ad8100ade72d2"}
Sep 18 19:50:53.747950 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:53.747912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:56.262559 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:56.262474    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerID="4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd" exitCode=0
Sep 18 19:50:56.262559 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:56.262524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerDied","Data":"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"}
Sep 18 19:50:56.831538 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:56.831504    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:50:56.831824 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:56.831793    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" containerID="cri-o://d7a8347708d89aa12938022788f46c11bc581af07258f75e9e8a9c2a0f326a9f" gracePeriod=30
Sep 18 19:50:56.831918 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:56.831878    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kube-rbac-proxy" containerID="cri-o://98c9e143f65967f735313ce61d3a198774cf9bb197b73049e488f790b13995bc" gracePeriod=30
Sep 18 19:50:57.269865 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.269830    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerID="98c9e143f65967f735313ce61d3a198774cf9bb197b73049e488f790b13995bc" exitCode=2
Sep 18 19:50:57.270332 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.269901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerDied","Data":"98c9e143f65967f735313ce61d3a198774cf9bb197b73049e488f790b13995bc"}
Sep 18 19:50:57.272402 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272375    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerStarted","Data":"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"}
Sep 18 19:50:57.272527 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerStarted","Data":"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"}
Sep 18 19:50:57.272527 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerStarted","Data":"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"}
Sep 18 19:50:57.272765 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272745    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:57.272842 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272811    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:57.272889 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.272847    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:50:57.274447 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.274421    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:50:57.275119 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.275098    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:57.294583 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:57.294534    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podStartSLOduration=3.421551012 podStartE2EDuration="7.29451792s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.872966908s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:57.293256637 +0000 UTC m=+765.811226276" watchObservedRunningTime="2026-09-18 19:50:57.29451792 +0000 UTC m=+765.812487561"
Sep 18 19:50:58.277170 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.277128    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:50:58.277599 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.277458    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:58.748130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.748089    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:50:58.752743 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.752711    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:50:58.752867 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.752843    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:50:58.752917 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.752838    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:5000: connect: connection refused"
Sep 18 19:50:58.752964 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:58.752952    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:50:59.903515 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:59.903469    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 19:50:59.907793 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:50:59.907768    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:51:00.757757 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.757733    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:51:00.855860 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.855814    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znzzw\" (UniqueName: \"kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw\") pod \"6d07102d-a257-49ea-b38d-1daea83ae64b\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") "
Sep 18 19:51:00.855860 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.855878    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert\") pod \"6d07102d-a257-49ea-b38d-1daea83ae64b\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") "
Sep 18 19:51:00.856118 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.855945    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location\") pod \"6d07102d-a257-49ea-b38d-1daea83ae64b\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") "
Sep 18 19:51:00.856118 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.855978    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config\") pod \"6d07102d-a257-49ea-b38d-1daea83ae64b\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") "
Sep 18 19:51:00.856118 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.856013    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls\") pod \"6d07102d-a257-49ea-b38d-1daea83ae64b\" (UID: \"6d07102d-a257-49ea-b38d-1daea83ae64b\") "
Sep 18 19:51:00.856331 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.856306    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d07102d-a257-49ea-b38d-1daea83ae64b" (UID: "6d07102d-a257-49ea-b38d-1daea83ae64b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:00.856372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.856353    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d07102d-a257-49ea-b38d-1daea83ae64b" (UID: "6d07102d-a257-49ea-b38d-1daea83ae64b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:00.856372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.856357    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d07102d-a257-49ea-b38d-1daea83ae64b" (UID: "6d07102d-a257-49ea-b38d-1daea83ae64b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:00.858007 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.857971    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw" (OuterVolumeSpecName: "kube-api-access-znzzw") pod "6d07102d-a257-49ea-b38d-1daea83ae64b" (UID: "6d07102d-a257-49ea-b38d-1daea83ae64b"). InnerVolumeSpecName "kube-api-access-znzzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:00.858102 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.858024    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d07102d-a257-49ea-b38d-1daea83ae64b" (UID: "6d07102d-a257-49ea-b38d-1daea83ae64b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:00.956667 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.956632    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d07102d-a257-49ea-b38d-1daea83ae64b-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:00.956667 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.956660    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzzw\" (UniqueName: \"kubernetes.io/projected/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-api-access-znzzw\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:00.956667 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.956670    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:00.957103 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.956678    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d07102d-a257-49ea-b38d-1daea83ae64b-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:00.957103 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:00.956688    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d07102d-a257-49ea-b38d-1daea83ae64b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:01.293375 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.293281    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerID="0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664" exitCode=0
Sep 18 19:51:01.293375 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.293351    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"
Sep 18 19:51:01.293375 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.293363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerDied","Data":"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"}
Sep 18 19:51:01.293649 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.293460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2" event={"ID":"6d07102d-a257-49ea-b38d-1daea83ae64b","Type":"ContainerDied","Data":"e7823b8866969bed60bc1a5d52a8e1d79de6604cbb0590d046562084b1afd518"}
Sep 18 19:51:01.293649 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.293491    2573 scope.go:117] "RemoveContainer" containerID="0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"
Sep 18 19:51:01.304973 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.304953    2573 scope.go:117] "RemoveContainer" containerID="f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"
Sep 18 19:51:01.312915 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.312903    2573 scope.go:117] "RemoveContainer" containerID="c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"
Sep 18 19:51:01.315683 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.315659    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:51:01.319609 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.319587    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-khjj2"]
Sep 18 19:51:01.321057 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.321043    2573 scope.go:117] "RemoveContainer" containerID="c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"
Sep 18 19:51:01.328480 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.328458    2573 scope.go:117] "RemoveContainer" containerID="0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"
Sep 18 19:51:01.328788 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:51:01.328770    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664\": container with ID starting with 0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664 not found: ID does not exist" containerID="0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"
Sep 18 19:51:01.328848 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.328795    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664"} err="failed to get container status \"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664\": rpc error: code = NotFound desc = could not find container \"0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664\": container with ID starting with 0a14b77d99df68a23a2a89dd2e837a9d001c7e32ecb788f306bc337f7efb3664 not found: ID does not exist"
Sep 18 19:51:01.328848 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.328812    2573 scope.go:117] "RemoveContainer" containerID="f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"
Sep 18 19:51:01.329082 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:51:01.329066    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1\": container with ID starting with f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1 not found: ID does not exist" containerID="f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"
Sep 18 19:51:01.329130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.329086    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1"} err="failed to get container status \"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1\": rpc error: code = NotFound desc = could not find container \"f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1\": container with ID starting with f3f2aedb48010d36f06a74c2c5f075007be952a9e7b74a4c7118761ac83963b1 not found: ID does not exist"
Sep 18 19:51:01.329130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.329100    2573 scope.go:117] "RemoveContainer" containerID="c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"
Sep 18 19:51:01.329399 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:51:01.329380    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d\": container with ID starting with c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d not found: ID does not exist" containerID="c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"
Sep 18 19:51:01.329467 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.329403    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d"} err="failed to get container status \"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d\": rpc error: code = NotFound desc = could not find container \"c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d\": container with ID starting with c2b3ddf91218044b1c2421102d098c6a69207da4e33749237b17ff904c7ee69d not found: ID does not exist"
Sep 18 19:51:01.329467 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.329417    2573 scope.go:117] "RemoveContainer" containerID="c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"
Sep 18 19:51:01.329702 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:51:01.329686    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa\": container with ID starting with c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa not found: ID does not exist" containerID="c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"
Sep 18 19:51:01.329749 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.329707    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa"} err="failed to get container status \"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa\": rpc error: code = NotFound desc = could not find container \"c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa\": container with ID starting with c7bd3eb633432f8f6e81ec52d65ba21689ba2e835164a5eb351ca3561e16d6fa not found: ID does not exist"
Sep 18 19:51:01.782066 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:01.782032    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" path="/var/lib/kubelet/pods/6d07102d-a257-49ea-b38d-1daea83ae64b/volumes"
Sep 18 19:51:02.023023 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.022986    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:51:02.023425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023407    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container"
Sep 18 19:51:02.023425 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023420    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023434    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="storage-initializer"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023440    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="storage-initializer"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023447    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023454    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023477    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy"
Sep 18 19:51:02.023502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023485    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy"
Sep 18 19:51:02.023679 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023552    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="agent"
Sep 18 19:51:02.023679 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023562    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kserve-container"
Sep 18 19:51:02.023679 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.023571    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d07102d-a257-49ea-b38d-1daea83ae64b" containerName="kube-rbac-proxy"
Sep 18 19:51:02.038426 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.038344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.039269 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.039247    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:51:02.040768 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.040750    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 18 19:51:02.040853 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.040789    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 19:51:02.167135 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.167096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.167135 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.167134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.167436 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.167159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.167436 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.167339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf29q\" (UniqueName: \"kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.167436 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.167417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268404 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf29q\" (UniqueName: \"kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268557 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.268945 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.268919    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.269277 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.269242    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.269395 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.269338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.271101 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.271076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.276610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.276582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf29q\" (UniqueName: \"kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.300788 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.300714    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerID="d7a8347708d89aa12938022788f46c11bc581af07258f75e9e8a9c2a0f326a9f" exitCode=0
Sep 18 19:51:02.300927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.300789    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerDied","Data":"d7a8347708d89aa12938022788f46c11bc581af07258f75e9e8a9c2a0f326a9f"}
Sep 18 19:51:02.348890 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.348855    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:02.387736 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.387669    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:51:02.470106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470074    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") pod \"9a198c14-f918-43d3-98ff-3e586474f0ba\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") "
Sep 18 19:51:02.470106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbtq\" (UniqueName: \"kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq\") pod \"9a198c14-f918-43d3-98ff-3e586474f0ba\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") "
Sep 18 19:51:02.470393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config\") pod \"9a198c14-f918-43d3-98ff-3e586474f0ba\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") "
Sep 18 19:51:02.470393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470151    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert\") pod \"9a198c14-f918-43d3-98ff-3e586474f0ba\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") "
Sep 18 19:51:02.470393 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470265    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location\") pod \"9a198c14-f918-43d3-98ff-3e586474f0ba\" (UID: \"9a198c14-f918-43d3-98ff-3e586474f0ba\") "
Sep 18 19:51:02.470595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470571    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a198c14-f918-43d3-98ff-3e586474f0ba" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:02.470595 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470582    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a198c14-f918-43d3-98ff-3e586474f0ba" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:02.470688 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.470595    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a198c14-f918-43d3-98ff-3e586474f0ba" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:02.472401 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.472379    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a198c14-f918-43d3-98ff-3e586474f0ba" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:02.472503 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.472428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq" (OuterVolumeSpecName: "kube-api-access-lzbtq") pod "9a198c14-f918-43d3-98ff-3e586474f0ba" (UID: "9a198c14-f918-43d3-98ff-3e586474f0ba"). InnerVolumeSpecName "kube-api-access-lzbtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:02.571027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.570944    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a198c14-f918-43d3-98ff-3e586474f0ba-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:02.571027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.570973    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a198c14-f918-43d3-98ff-3e586474f0ba-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:02.571027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.570985    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbtq\" (UniqueName: \"kubernetes.io/projected/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-api-access-lzbtq\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:02.571027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.570996    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:02.571027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.571005    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a198c14-f918-43d3-98ff-3e586474f0ba-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:51:02.622338 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:02.622312    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:51:02.624840 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:51:02.624803    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d2a18a_0101_4c02_b078_7876b6ec7178.slice/crio-2a59a236b826847d649c9f15155f6e6aec4914597e53859868edb4b155af4da1 WatchSource:0}: Error finding container 2a59a236b826847d649c9f15155f6e6aec4914597e53859868edb4b155af4da1: Status 404 returned error can't find the container with id 2a59a236b826847d649c9f15155f6e6aec4914597e53859868edb4b155af4da1
Sep 18 19:51:03.282512 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.282487    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:51:03.283027 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.283003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:03.283583 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.283561    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:03.308074 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.308048    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"
Sep 18 19:51:03.308288 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.308079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt" event={"ID":"9a198c14-f918-43d3-98ff-3e586474f0ba","Type":"ContainerDied","Data":"266e179801c909f4f6d63ccaa7e907e8796eae9d8bfa49477f24278cff3efda1"}
Sep 18 19:51:03.308288 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.308167    2573 scope.go:117] "RemoveContainer" containerID="98c9e143f65967f735313ce61d3a198774cf9bb197b73049e488f790b13995bc"
Sep 18 19:51:03.309824 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.309802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerStarted","Data":"82384b2bc5ffe30631d26446b6ddd4e8c8befded56fb19f79920dafa4c26b30a"}
Sep 18 19:51:03.309937 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.309854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerStarted","Data":"2a59a236b826847d649c9f15155f6e6aec4914597e53859868edb4b155af4da1"}
Sep 18 19:51:03.317642 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.317628    2573 scope.go:117] "RemoveContainer" containerID="d7a8347708d89aa12938022788f46c11bc581af07258f75e9e8a9c2a0f326a9f"
Sep 18 19:51:03.325123 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.325103    2573 scope.go:117] "RemoveContainer" containerID="87b5c152f126a2fec1e2bdf19180c8cf848c6621e7870d3d7a3bafe87fdb0814"
Sep 18 19:51:03.338527 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.338502    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:51:03.342444 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.342422    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-whrdt"]
Sep 18 19:51:03.782544 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:03.782507    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" path="/var/lib/kubelet/pods/9a198c14-f918-43d3-98ff-3e586474f0ba/volumes"
Sep 18 19:51:07.328988 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:07.328948    2573 generic.go:356] "Generic (PLEG): container finished" podID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerID="82384b2bc5ffe30631d26446b6ddd4e8c8befded56fb19f79920dafa4c26b30a" exitCode=0
Sep 18 19:51:07.329403 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:07.329032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerDied","Data":"82384b2bc5ffe30631d26446b6ddd4e8c8befded56fb19f79920dafa4c26b30a"}
Sep 18 19:51:08.337292 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.337251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerStarted","Data":"c63211cb8d4468ae895891a3a5e2bd743f4bf8b59b02fe8ed2283408192e6e35"}
Sep 18 19:51:08.337734 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.337320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerStarted","Data":"7f56eccdcc45605d6aa19aafdf0b43b8dbce0a0de996585f7b6c1025815061cb"}
Sep 18 19:51:08.337734 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.337587    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:08.337816 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.337734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:08.339004 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.338979    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:08.355407 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:08.355365    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podStartSLOduration=2.396785552 podStartE2EDuration="6.355352493s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.958566941s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:51:08.354506278 +0000 UTC m=+776.872475920" watchObservedRunningTime="2026-09-18 19:51:08.355352493 +0000 UTC m=+776.873322133"
Sep 18 19:51:09.342081 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:09.342046    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:13.283690 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:13.283638    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:13.284127 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:13.284076    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:14.346722 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:14.346689    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:51:14.347251 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:14.347224    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:23.282986 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:23.282942    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:23.283437 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:23.283362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:24.348015 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:24.347977    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:33.283559 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:33.283478    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:33.284029 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:33.284009    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:34.347445 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:34.347407    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:43.283280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:43.283243    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:43.283875 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:43.283848    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:44.347373 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:44.347332    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:51:53.283003 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:53.282952    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:51:53.283491 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:53.283401    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:51:54.347861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:51:54.347818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:52:03.283371 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:03.283325    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:52:03.283887 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:03.283869    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:04.347225 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:04.347165    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:52:13.283958 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:13.283929    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:13.284372 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:13.284261    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:14.347926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:14.347884    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:52:24.347416 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:24.347374    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:52:25.764975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.764947    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-65ngk_abf6a7b2-2ead-49ba-ac48-850a8e95b3ab/kserve-container/0.log"
Sep 18 19:52:25.918385 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.918342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:52:25.918727 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.918694    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" containerID="cri-o://45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c" gracePeriod=30
Sep 18 19:52:25.918954 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.918871    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" containerID="cri-o://6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4" gracePeriod=30
Sep 18 19:52:25.919062 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.918938    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" containerID="cri-o://77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d" gracePeriod=30
Sep 18 19:52:25.926926 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.926901    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:52:25.928024 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.927161    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kserve-container" containerID="cri-o://789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5" gracePeriod=30
Sep 18 19:52:25.928024 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:25.927240    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kube-rbac-proxy" containerID="cri-o://829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040" gracePeriod=30
Sep 18 19:52:26.033839 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.033763    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:52:26.034238 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034216    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kube-rbac-proxy"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034241    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kube-rbac-proxy"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034261    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="storage-initializer"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034270    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="storage-initializer"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034291    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034297    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container"
Sep 18 19:52:26.034358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034359    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kserve-container"
Sep 18 19:52:26.034964 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.034368    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a198c14-f918-43d3-98ff-3e586474f0ba" containerName="kube-rbac-proxy"
Sep 18 19:52:26.040413 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.040393    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.042832 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.042809    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 18 19:52:26.042935 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.042818    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 18 19:52:26.047512 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.047487    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:52:26.095585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.095544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.095585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.095586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.095778 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.095621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.095778 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.095661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.095778 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.095744    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j64\" (UniqueName: \"kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.183586 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.183562    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:52:26.197193 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197298 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197298 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j64\" (UniqueName: \"kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197408 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197500 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:26.197458    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 18 19:52:26.197541 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:26.197534    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls podName:7cdd88d0-eddc-4bff-b224-1c5897d7e9fc nodeName:}" failed. No retries permitted until 2026-09-18 19:52:26.697516941 +0000 UTC m=+855.215486575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-df69n" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 18 19:52:26.197719 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197880 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197862    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.197921 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.197878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.208135 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.208109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j64\" (UniqueName: \"kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.297767 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.297691    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7plxf\" (UniqueName: \"kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf\") pod \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") "
Sep 18 19:52:26.297767 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.297730    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config\") pod \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") "
Sep 18 19:52:26.297941 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.297784    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") pod \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\" (UID: \"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab\") "
Sep 18 19:52:26.298187 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.298158    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" (UID: "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:26.299812 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.299791    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" (UID: "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:26.299907 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.299844    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf" (OuterVolumeSpecName: "kube-api-access-7plxf") pod "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" (UID: "abf6a7b2-2ead-49ba-ac48-850a8e95b3ab"). InnerVolumeSpecName "kube-api-access-7plxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:26.398349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.398306    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7plxf\" (UniqueName: \"kubernetes.io/projected/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-api-access-7plxf\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:26.398349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.398342    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:26.398349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.398355    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:26.669103 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669068    2573 generic.go:356] "Generic (PLEG): container finished" podID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerID="829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040" exitCode=2
Sep 18 19:52:26.669103 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669094    2573 generic.go:356] "Generic (PLEG): container finished" podID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerID="789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5" exitCode=2
Sep 18 19:52:26.669353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669121    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"
Sep 18 19:52:26.669353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerDied","Data":"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"}
Sep 18 19:52:26.669353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerDied","Data":"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"}
Sep 18 19:52:26.669353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk" event={"ID":"abf6a7b2-2ead-49ba-ac48-850a8e95b3ab","Type":"ContainerDied","Data":"94d22ff85f793c7822402959f0db9b1e8b7cf4fe974d9038ad9e283f80a71003"}
Sep 18 19:52:26.669353 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.669344    2573 scope.go:117] "RemoveContainer" containerID="829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"
Sep 18 19:52:26.672101 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.672080    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerID="6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4" exitCode=2
Sep 18 19:52:26.672233 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.672116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerDied","Data":"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"}
Sep 18 19:52:26.678863 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.678844    2573 scope.go:117] "RemoveContainer" containerID="789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"
Sep 18 19:52:26.686321 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.686225    2573 scope.go:117] "RemoveContainer" containerID="829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"
Sep 18 19:52:26.686540 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:26.686520    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040\": container with ID starting with 829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040 not found: ID does not exist" containerID="829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"
Sep 18 19:52:26.686618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.686550    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"} err="failed to get container status \"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040\": rpc error: code = NotFound desc = could not find container \"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040\": container with ID starting with 829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040 not found: ID does not exist"
Sep 18 19:52:26.686618 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.686573    2573 scope.go:117] "RemoveContainer" containerID="789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"
Sep 18 19:52:26.686841 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:26.686822    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5\": container with ID starting with 789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5 not found: ID does not exist" containerID="789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"
Sep 18 19:52:26.686891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.686845    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"} err="failed to get container status \"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5\": rpc error: code = NotFound desc = could not find container \"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5\": container with ID starting with 789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5 not found: ID does not exist"
Sep 18 19:52:26.686891 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.686860    2573 scope.go:117] "RemoveContainer" containerID="829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"
Sep 18 19:52:26.687092 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.687077    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040"} err="failed to get container status \"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040\": rpc error: code = NotFound desc = could not find container \"829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040\": container with ID starting with 829b82c75b83bc2f7f392fb7a412a9b3fe8a130eb5e9ad11d20b301082195040 not found: ID does not exist"
Sep 18 19:52:26.687133 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.687091    2573 scope.go:117] "RemoveContainer" containerID="789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"
Sep 18 19:52:26.687365 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.687346    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5"} err="failed to get container status \"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5\": rpc error: code = NotFound desc = could not find container \"789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5\": container with ID starting with 789cd533884faa626b4761cb68030376bd63740cf7ea1514c0feab700c26c3a5 not found: ID does not exist"
Sep 18 19:52:26.691424 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.691402    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:52:26.696579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.696558    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-65ngk"]
Sep 18 19:52:26.700325 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.700308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.702533 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.702515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-df69n\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:26.952585 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:26.952549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:52:27.222402 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:27.222376    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:52:27.224833 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:52:27.224808    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdd88d0_eddc_4bff_b224_1c5897d7e9fc.slice/crio-475341870f3e259769f4aadb4f4254ea9e55f1521cda4bfbe8bf8729d56aee79 WatchSource:0}: Error finding container 475341870f3e259769f4aadb4f4254ea9e55f1521cda4bfbe8bf8729d56aee79: Status 404 returned error can't find the container with id 475341870f3e259769f4aadb4f4254ea9e55f1521cda4bfbe8bf8729d56aee79
Sep 18 19:52:27.680194 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:27.680153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerStarted","Data":"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"}
Sep 18 19:52:27.680414 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:27.680231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerStarted","Data":"475341870f3e259769f4aadb4f4254ea9e55f1521cda4bfbe8bf8729d56aee79"}
Sep 18 19:52:27.783535 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:27.783498    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" path="/var/lib/kubelet/pods/abf6a7b2-2ead-49ba-ac48-850a8e95b3ab/volumes"
Sep 18 19:52:28.278279 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:28.278234    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:31.700760 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:31.700729    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerID="45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c" exitCode=0
Sep 18 19:52:31.701130 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:31.700814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerDied","Data":"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"}
Sep 18 19:52:32.707080 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:32.707043    2573 generic.go:356] "Generic (PLEG): container finished" podID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerID="d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08" exitCode=0
Sep 18 19:52:32.707478 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:32.707122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerDied","Data":"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"}
Sep 18 19:52:33.278217 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:33.278160    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:33.283162 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:33.283040    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:52:33.283591 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:33.283562    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:34.348131 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:34.348065    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:52:38.278040 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:38.278001    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:38.278567 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:38.278126    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:42.826706 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:42.826509    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:52:42.827356 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:42.826832    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" containerID="cri-o://7f56eccdcc45605d6aa19aafdf0b43b8dbce0a0de996585f7b6c1025815061cb" gracePeriod=30
Sep 18 19:52:42.827356 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:42.827011    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kube-rbac-proxy" containerID="cri-o://c63211cb8d4468ae895891a3a5e2bd743f4bf8b59b02fe8ed2283408192e6e35" gracePeriod=30
Sep 18 19:52:43.277341 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:43.277286    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:43.283526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:43.283494    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:52:43.284040 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:43.284004    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:43.768234 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:43.768179    2573 generic.go:356] "Generic (PLEG): container finished" podID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerID="c63211cb8d4468ae895891a3a5e2bd743f4bf8b59b02fe8ed2283408192e6e35" exitCode=2
Sep 18 19:52:43.768406 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:43.768253    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerDied","Data":"c63211cb8d4468ae895891a3a5e2bd743f4bf8b59b02fe8ed2283408192e6e35"}
Sep 18 19:52:44.342757 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:44.342710    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 19:52:44.348141 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:44.348105    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:52:48.277674 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:48.277629    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:48.798936 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:48.798898    2573 generic.go:356] "Generic (PLEG): container finished" podID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerID="7f56eccdcc45605d6aa19aafdf0b43b8dbce0a0de996585f7b6c1025815061cb" exitCode=0
Sep 18 19:52:48.799097 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:48.798992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerDied","Data":"7f56eccdcc45605d6aa19aafdf0b43b8dbce0a0de996585f7b6c1025815061cb"}
Sep 18 19:52:49.015987 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.015957    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:52:49.105253 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105220    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls\") pod \"21d2a18a-0101-4c02-b078-7876b6ec7178\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") "
Sep 18 19:52:49.105253 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location\") pod \"21d2a18a-0101-4c02-b078-7876b6ec7178\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") "
Sep 18 19:52:49.105516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105291    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf29q\" (UniqueName: \"kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q\") pod \"21d2a18a-0101-4c02-b078-7876b6ec7178\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") "
Sep 18 19:52:49.105516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105356    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert\") pod \"21d2a18a-0101-4c02-b078-7876b6ec7178\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") "
Sep 18 19:52:49.105516 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105420    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config\") pod \"21d2a18a-0101-4c02-b078-7876b6ec7178\" (UID: \"21d2a18a-0101-4c02-b078-7876b6ec7178\") "
Sep 18 19:52:49.105691 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105655    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21d2a18a-0101-4c02-b078-7876b6ec7178" (UID: "21d2a18a-0101-4c02-b078-7876b6ec7178"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:49.106016 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105985    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21d2a18a-0101-4c02-b078-7876b6ec7178" (UID: "21d2a18a-0101-4c02-b078-7876b6ec7178"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:49.106016 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.105983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21d2a18a-0101-4c02-b078-7876b6ec7178" (UID: "21d2a18a-0101-4c02-b078-7876b6ec7178"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:49.107821 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.107792    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q" (OuterVolumeSpecName: "kube-api-access-wf29q") pod "21d2a18a-0101-4c02-b078-7876b6ec7178" (UID: "21d2a18a-0101-4c02-b078-7876b6ec7178"). InnerVolumeSpecName "kube-api-access-wf29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:49.107930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.107887    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21d2a18a-0101-4c02-b078-7876b6ec7178" (UID: "21d2a18a-0101-4c02-b078-7876b6ec7178"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:49.206659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.206578    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:49.206659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.206608    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d2a18a-0101-4c02-b078-7876b6ec7178-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:49.206659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.206619    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d2a18a-0101-4c02-b078-7876b6ec7178-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:49.206659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.206628    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf29q\" (UniqueName: \"kubernetes.io/projected/21d2a18a-0101-4c02-b078-7876b6ec7178-kube-api-access-wf29q\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:49.206659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.206637    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d2a18a-0101-4c02-b078-7876b6ec7178-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:49.804784 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.804760    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"
Sep 18 19:52:49.805139 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.804777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6" event={"ID":"21d2a18a-0101-4c02-b078-7876b6ec7178","Type":"ContainerDied","Data":"2a59a236b826847d649c9f15155f6e6aec4914597e53859868edb4b155af4da1"}
Sep 18 19:52:49.805139 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.804843    2573 scope.go:117] "RemoveContainer" containerID="c63211cb8d4468ae895891a3a5e2bd743f4bf8b59b02fe8ed2283408192e6e35"
Sep 18 19:52:49.813847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.813827    2573 scope.go:117] "RemoveContainer" containerID="7f56eccdcc45605d6aa19aafdf0b43b8dbce0a0de996585f7b6c1025815061cb"
Sep 18 19:52:49.823148 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.823118    2573 scope.go:117] "RemoveContainer" containerID="82384b2bc5ffe30631d26446b6ddd4e8c8befded56fb19f79920dafa4c26b30a"
Sep 18 19:52:49.825256 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.825228    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:52:49.827110 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:49.827087    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-j4rp6"]
Sep 18 19:52:51.784995 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:51.784956    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" path="/var/lib/kubelet/pods/21d2a18a-0101-4c02-b078-7876b6ec7178/volumes"
Sep 18 19:52:53.277982 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:53.277933    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:52:53.283377 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:53.283326    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:52:53.283539 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:53.283502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:53.283854 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:53.283821    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:52:53.283994 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:53.283974    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:56.146496 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.146458    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:56.273510 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273426    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location\") pod \"bfbc3440-4645-4483-bf41-67c7ee10f761\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") "
Sep 18 19:52:56.273510 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273462    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config\") pod \"bfbc3440-4645-4483-bf41-67c7ee10f761\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") "
Sep 18 19:52:56.273510 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trmqr\" (UniqueName: \"kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr\") pod \"bfbc3440-4645-4483-bf41-67c7ee10f761\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") "
Sep 18 19:52:56.273801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273589    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") pod \"bfbc3440-4645-4483-bf41-67c7ee10f761\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") "
Sep 18 19:52:56.273801 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273700    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert\") pod \"bfbc3440-4645-4483-bf41-67c7ee10f761\" (UID: \"bfbc3440-4645-4483-bf41-67c7ee10f761\") "
Sep 18 19:52:56.273912 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273817    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfbc3440-4645-4483-bf41-67c7ee10f761" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:56.273912 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273855    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfbc3440-4645-4483-bf41-67c7ee10f761" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:56.274031 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273965    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfbc3440-4645-4483-bf41-67c7ee10f761-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.274031 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.273981    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.274354 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.274316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bfbc3440-4645-4483-bf41-67c7ee10f761" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:56.276264 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.276236    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr" (OuterVolumeSpecName: "kube-api-access-trmqr") pod "bfbc3440-4645-4483-bf41-67c7ee10f761" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761"). InnerVolumeSpecName "kube-api-access-trmqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:56.276423 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.276397    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfbc3440-4645-4483-bf41-67c7ee10f761" (UID: "bfbc3440-4645-4483-bf41-67c7ee10f761"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:56.374847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.374813    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmqr\" (UniqueName: \"kubernetes.io/projected/bfbc3440-4645-4483-bf41-67c7ee10f761-kube-api-access-trmqr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.374847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.374843    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbc3440-4645-4483-bf41-67c7ee10f761-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.374847 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.374852    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfbc3440-4645-4483-bf41-67c7ee10f761-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:52:56.842817 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.842776    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerID="77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d" exitCode=0
Sep 18 19:52:56.843008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.842861    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerDied","Data":"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"}
Sep 18 19:52:56.843008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.842932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh" event={"ID":"bfbc3440-4645-4483-bf41-67c7ee10f761","Type":"ContainerDied","Data":"58ee24c59317aaf184514bec67a85f99f0d5b455f0babaf15e9ad8100ade72d2"}
Sep 18 19:52:56.843008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.842868    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"
Sep 18 19:52:56.843008 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.842962    2573 scope.go:117] "RemoveContainer" containerID="77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"
Sep 18 19:52:56.853430 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.853408    2573 scope.go:117] "RemoveContainer" containerID="6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"
Sep 18 19:52:56.861526 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.861497    2573 scope.go:117] "RemoveContainer" containerID="45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"
Sep 18 19:52:56.867371 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.867349    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:52:56.870901 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.870760    2573 scope.go:117] "RemoveContainer" containerID="4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"
Sep 18 19:52:56.873191 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.873166    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-57qgh"]
Sep 18 19:52:56.880282 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.880258    2573 scope.go:117] "RemoveContainer" containerID="77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"
Sep 18 19:52:56.880622 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:56.880599    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d\": container with ID starting with 77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d not found: ID does not exist" containerID="77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"
Sep 18 19:52:56.880712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.880629    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d"} err="failed to get container status \"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d\": rpc error: code = NotFound desc = could not find container \"77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d\": container with ID starting with 77a1ffe86d9ed63f72e53fdcfb39596b30eaade6a3275e0edf1b56a20a8a1a4d not found: ID does not exist"
Sep 18 19:52:56.880712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.880648    2573 scope.go:117] "RemoveContainer" containerID="6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"
Sep 18 19:52:56.880939 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:56.880918    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4\": container with ID starting with 6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4 not found: ID does not exist" containerID="6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"
Sep 18 19:52:56.881013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.880957    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4"} err="failed to get container status \"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4\": rpc error: code = NotFound desc = could not find container \"6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4\": container with ID starting with 6bae7f81ced229dedb6dc9d24d1f90bc3f5860446966cdd8944e98b9344e9ec4 not found: ID does not exist"
Sep 18 19:52:56.881013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.880974    2573 scope.go:117] "RemoveContainer" containerID="45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"
Sep 18 19:52:56.881330 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:56.881309    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c\": container with ID starting with 45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c not found: ID does not exist" containerID="45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"
Sep 18 19:52:56.881407 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.881334    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c"} err="failed to get container status \"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c\": rpc error: code = NotFound desc = could not find container \"45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c\": container with ID starting with 45cbb60bb8c035c5e259b4e804ee6f3bafca66a7e920db1af74f55e6ab26c56c not found: ID does not exist"
Sep 18 19:52:56.881407 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.881349    2573 scope.go:117] "RemoveContainer" containerID="4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"
Sep 18 19:52:56.881627 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:52:56.881610    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd\": container with ID starting with 4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd not found: ID does not exist" containerID="4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"
Sep 18 19:52:56.881693 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:56.881629    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd"} err="failed to get container status \"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd\": rpc error: code = NotFound desc = could not find container \"4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd\": container with ID starting with 4600bf6a4741f0257986fac1ecb7f933b0109d0f8f2ada327fbbf5998d1c91bd not found: ID does not exist"
Sep 18 19:52:57.784270 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:52:57.784233    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" path="/var/lib/kubelet/pods/bfbc3440-4645-4483-bf41-67c7ee10f761/volumes"
Sep 18 19:54:29.280098 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.280060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerStarted","Data":"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"}
Sep 18 19:54:29.280579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.280123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerStarted","Data":"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"}
Sep 18 19:54:29.280579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.280158    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:29.280579 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.280187    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:29.281379 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.281357    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:54:29.298973 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:29.298920    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podStartSLOduration=2.626336607 podStartE2EDuration="2m3.298904217s" totalImagesPullingTime="1m55.840272185s" totalInitContainerRuntime="4.832295425s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:29.298314019 +0000 UTC m=+977.816283650" watchObservedRunningTime="2026-09-18 19:54:29.298904217 +0000 UTC m=+977.816873848"
Sep 18 19:54:30.284811 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:30.284770    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:54:35.290895 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:35.290866    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:35.291422 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:35.291326    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:54:45.292365 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:45.292325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:52.319502 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:52.319463    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:54:52.319998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:52.319718    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container" containerID="cri-o://2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa" gracePeriod=30
Sep 18 19:54:52.319998 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:52.319761    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kube-rbac-proxy" containerID="cri-o://4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14" gracePeriod=30
Sep 18 19:54:53.384056 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.384020    2573 generic.go:356] "Generic (PLEG): container finished" podID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerID="4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14" exitCode=2
Sep 18 19:54:53.384465 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.384062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerDied","Data":"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"}
Sep 18 19:54:53.881737 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.881709    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:53.995576 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.995492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config\") pod \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") "
Sep 18 19:54:53.995576 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.995529    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7j64\" (UniqueName: \"kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64\") pod \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") "
Sep 18 19:54:53.995818 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.995580    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") pod \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") "
Sep 18 19:54:53.995818 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.995609    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location\") pod \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") "
Sep 18 19:54:53.995818 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.995653    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert\") pod \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\" (UID: \"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc\") "
Sep 18 19:54:53.996037 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.996000    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:53.996124 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.996098    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:53.997845 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.997816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:53.997959 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:53.997839    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64" (OuterVolumeSpecName: "kube-api-access-s7j64") pod "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc"). InnerVolumeSpecName "kube-api-access-s7j64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:54.000184 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.000160    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" (UID: "7cdd88d0-eddc-4bff-b224-1c5897d7e9fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:54.096927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.096888    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:54:54.096927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.096921    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7j64\" (UniqueName: \"kubernetes.io/projected/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kube-api-access-s7j64\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:54:54.096927 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.096934    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:54:54.097180 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.096945    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:54:54.097180 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.096957    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:54:54.390975 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.390938    2573 generic.go:356] "Generic (PLEG): container finished" podID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerID="2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa" exitCode=0
Sep 18 19:54:54.391487 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.391006    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"
Sep 18 19:54:54.391487 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.391024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerDied","Data":"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"}
Sep 18 19:54:54.391487 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.391091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n" event={"ID":"7cdd88d0-eddc-4bff-b224-1c5897d7e9fc","Type":"ContainerDied","Data":"475341870f3e259769f4aadb4f4254ea9e55f1521cda4bfbe8bf8729d56aee79"}
Sep 18 19:54:54.391487 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.391119    2573 scope.go:117] "RemoveContainer" containerID="4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"
Sep 18 19:54:54.401513 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.401492    2573 scope.go:117] "RemoveContainer" containerID="2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"
Sep 18 19:54:54.413564 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.413540    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:54:54.414519 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.414500    2573 scope.go:117] "RemoveContainer" containerID="d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"
Sep 18 19:54:54.418857 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.418835    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-df69n"]
Sep 18 19:54:54.422327 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.422307    2573 scope.go:117] "RemoveContainer" containerID="4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"
Sep 18 19:54:54.422641 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:54:54.422621    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14\": container with ID starting with 4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14 not found: ID does not exist" containerID="4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"
Sep 18 19:54:54.422693 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.422649    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14"} err="failed to get container status \"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14\": rpc error: code = NotFound desc = could not find container \"4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14\": container with ID starting with 4b31989b92a77196ede4cd6b11b057f10ee3c133ea7160a3f7a0429a0715dd14 not found: ID does not exist"
Sep 18 19:54:54.422693 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.422667    2573 scope.go:117] "RemoveContainer" containerID="2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"
Sep 18 19:54:54.422931 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:54:54.422913    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa\": container with ID starting with 2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa not found: ID does not exist" containerID="2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"
Sep 18 19:54:54.422973 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.422937    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa"} err="failed to get container status \"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa\": rpc error: code = NotFound desc = could not find container \"2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa\": container with ID starting with 2ebcd3f5698cbb23acc82e77ebed93b66d1ec9fd7016c8c8e37bfc7d737011fa not found: ID does not exist"
Sep 18 19:54:54.422973 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.422953    2573 scope.go:117] "RemoveContainer" containerID="d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"
Sep 18 19:54:54.423174 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:54:54.423160    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08\": container with ID starting with d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08 not found: ID does not exist" containerID="d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"
Sep 18 19:54:54.423237 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:54.423178    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08"} err="failed to get container status \"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08\": rpc error: code = NotFound desc = could not find container \"d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08\": container with ID starting with d462e8519b6a164b8bfba2df0d26670c8207e4986c9de1ddeb95ba81fbf1fe08 not found: ID does not exist"
Sep 18 19:54:55.783307 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:55.783274    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" path="/var/lib/kubelet/pods/7cdd88d0-eddc-4bff-b224-1c5897d7e9fc/volumes"
Sep 18 19:54:57.486448 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486407    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486742    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486763    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kserve-container"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486769    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kserve-container"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kube-rbac-proxy"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kube-rbac-proxy"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486790    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486795    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486800    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486805    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486813    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="storage-initializer"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486818    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="storage-initializer"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486826    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="storage-initializer"
Sep 18 19:54:57.486834 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486834    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="storage-initializer"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486844    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486851    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486858    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="storage-initializer"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486863    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="storage-initializer"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486869    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486874    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486889    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486894    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486960    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486968    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="agent"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486974    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486981    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486989    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.486999    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cdd88d0-eddc-4bff-b224-1c5897d7e9fc" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.487005    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf6a7b2-2ead-49ba-ac48-850a8e95b3ab" containerName="kserve-container"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.487010    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d2a18a-0101-4c02-b078-7876b6ec7178" containerName="kube-rbac-proxy"
Sep 18 19:54:57.487297 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.487015    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbc3440-4645-4483-bf41-67c7ee10f761" containerName="kube-rbac-proxy"
Sep 18 19:54:57.517916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.517885    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:54:57.518334 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.518013    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.520257 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.520240    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:54:57.520563 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.520550    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:54:57.520621 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.520606    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 18 19:54:57.520823 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.520803    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:54:57.521268 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.521249    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 19:54:57.521268 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.521260    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:54:57.521415 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.521278    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:54:57.628919 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.628882    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vl29\" (UniqueName: \"kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.629100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.628932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.629100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.629005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.629100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.629043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.629100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.629080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.729598 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.729563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vl29\" (UniqueName: \"kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.729781 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.729621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.729781 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.729650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.729781 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.729675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.729781 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.729705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.730183 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.730159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.730345 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.730326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.730411 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.730395    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.732268 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.732248    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.737349 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.737286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vl29\" (UniqueName: \"kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-thssc\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:57.828886 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:57.828855    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:54:58.114100 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:58.114041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:54:58.116315 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:54:58.116284    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4863ecbc_4a7d_4123_924a_c720779cd0ee.slice/crio-465268d4e1db6e75fcf566204393390568ba1f9ede946cce53fc78998a5c7936 WatchSource:0}: Error finding container 465268d4e1db6e75fcf566204393390568ba1f9ede946cce53fc78998a5c7936: Status 404 returned error can't find the container with id 465268d4e1db6e75fcf566204393390568ba1f9ede946cce53fc78998a5c7936
Sep 18 19:54:58.409610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:58.409520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerStarted","Data":"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"}
Sep 18 19:54:58.409610 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:54:58.409596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerStarted","Data":"465268d4e1db6e75fcf566204393390568ba1f9ede946cce53fc78998a5c7936"}
Sep 18 19:55:19.500013 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:19.499977    2573 generic.go:356] "Generic (PLEG): container finished" podID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerID="4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041" exitCode=0
Sep 18 19:55:19.500453 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:19.500044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerDied","Data":"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"}
Sep 18 19:55:20.506571 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:20.506536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerStarted","Data":"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"}
Sep 18 19:55:20.506976 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:20.506601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerStarted","Data":"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"}
Sep 18 19:55:20.506976 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:20.506936    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:55:20.507070 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:20.506989    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:55:20.526420 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:20.526363    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" podStartSLOduration=2.852890357 podStartE2EDuration="23.526348439s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.673458082s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:20.524292062 +0000 UTC m=+1029.042261704" watchObservedRunningTime="2026-09-18 19:55:20.526348439 +0000 UTC m=+1029.044318078"
Sep 18 19:55:26.515562 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:55:26.515532    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:56:52.810861 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:56:52.810822    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:57:03.756280 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:03.756179    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:57:03.756837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:03.756507    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kserve-container" containerID="cri-o://57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939" gracePeriod=30
Sep 18 19:57:03.756837 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:03.756547    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kube-rbac-proxy" containerID="cri-o://5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319" gracePeriod=30
Sep 18 19:57:03.922543 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:03.922508    2573 generic.go:356] "Generic (PLEG): container finished" podID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerID="5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319" exitCode=2
Sep 18 19:57:03.922704 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:03.922576    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerDied","Data":"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"}
Sep 18 19:57:05.294740 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.294718    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:57:05.316643 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.316616    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location\") pod \"4863ecbc-4a7d-4123-924a-c720779cd0ee\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") "
Sep 18 19:57:05.316802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.316663    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config\") pod \"4863ecbc-4a7d-4123-924a-c720779cd0ee\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") "
Sep 18 19:57:05.316802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.316693    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls\") pod \"4863ecbc-4a7d-4123-924a-c720779cd0ee\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") "
Sep 18 19:57:05.316802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.316719    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert\") pod \"4863ecbc-4a7d-4123-924a-c720779cd0ee\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") "
Sep 18 19:57:05.317004 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.316978    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4863ecbc-4a7d-4123-924a-c720779cd0ee" (UID: "4863ecbc-4a7d-4123-924a-c720779cd0ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:05.317293 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.317262    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4863ecbc-4a7d-4123-924a-c720779cd0ee" (UID: "4863ecbc-4a7d-4123-924a-c720779cd0ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:05.318973 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.318948    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4863ecbc-4a7d-4123-924a-c720779cd0ee" (UID: "4863ecbc-4a7d-4123-924a-c720779cd0ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:05.321869 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.321843    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4863ecbc-4a7d-4123-924a-c720779cd0ee" (UID: "4863ecbc-4a7d-4123-924a-c720779cd0ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:05.417655 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.417551    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vl29\" (UniqueName: \"kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29\") pod \"4863ecbc-4a7d-4123-924a-c720779cd0ee\" (UID: \"4863ecbc-4a7d-4123-924a-c720779cd0ee\") "
Sep 18 19:57:05.417831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.417745    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4863ecbc-4a7d-4123-924a-c720779cd0ee-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:57:05.417831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.417764    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:57:05.417831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.417778    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4863ecbc-4a7d-4123-924a-c720779cd0ee-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:57:05.417831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.417790    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4863ecbc-4a7d-4123-924a-c720779cd0ee-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:57:05.419643 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.419617    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29" (OuterVolumeSpecName: "kube-api-access-5vl29") pod "4863ecbc-4a7d-4123-924a-c720779cd0ee" (UID: "4863ecbc-4a7d-4123-924a-c720779cd0ee"). InnerVolumeSpecName "kube-api-access-5vl29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:05.518733 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.518699    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vl29\" (UniqueName: \"kubernetes.io/projected/4863ecbc-4a7d-4123-924a-c720779cd0ee-kube-api-access-5vl29\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:57:05.932965 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.932932    2573 generic.go:356] "Generic (PLEG): container finished" podID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerID="57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939" exitCode=0
Sep 18 19:57:05.933158 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.932994    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"
Sep 18 19:57:05.933158 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.933023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerDied","Data":"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"}
Sep 18 19:57:05.933158 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.933087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc" event={"ID":"4863ecbc-4a7d-4123-924a-c720779cd0ee","Type":"ContainerDied","Data":"465268d4e1db6e75fcf566204393390568ba1f9ede946cce53fc78998a5c7936"}
Sep 18 19:57:05.933158 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.933122    2573 scope.go:117] "RemoveContainer" containerID="5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"
Sep 18 19:57:05.941600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.941583    2573 scope.go:117] "RemoveContainer" containerID="57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"
Sep 18 19:57:05.949512 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.949495    2573 scope.go:117] "RemoveContainer" containerID="4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"
Sep 18 19:57:05.950228 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.950192    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:57:05.955599 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.955580    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-thssc"]
Sep 18 19:57:05.957419 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.957401    2573 scope.go:117] "RemoveContainer" containerID="5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"
Sep 18 19:57:05.957686 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:57:05.957666    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319\": container with ID starting with 5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319 not found: ID does not exist" containerID="5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"
Sep 18 19:57:05.957730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.957702    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319"} err="failed to get container status \"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319\": rpc error: code = NotFound desc = could not find container \"5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319\": container with ID starting with 5c808ad6375137673cc4e8180a0e090e973743afcdbc3f19d3499f6345b63319 not found: ID does not exist"
Sep 18 19:57:05.957730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.957719    2573 scope.go:117] "RemoveContainer" containerID="57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"
Sep 18 19:57:05.957980 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:57:05.957962    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939\": container with ID starting with 57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939 not found: ID does not exist" containerID="57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"
Sep 18 19:57:05.958021 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.957987    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939"} err="failed to get container status \"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939\": rpc error: code = NotFound desc = could not find container \"57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939\": container with ID starting with 57cd4405551a515c3102559a71eb7108e55851396196955f9823281ef9d47939 not found: ID does not exist"
Sep 18 19:57:05.958021 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.958004    2573 scope.go:117] "RemoveContainer" containerID="4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"
Sep 18 19:57:05.958260 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:57:05.958242    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041\": container with ID starting with 4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041 not found: ID does not exist" containerID="4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"
Sep 18 19:57:05.958305 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:05.958268    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041"} err="failed to get container status \"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041\": rpc error: code = NotFound desc = could not find container \"4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041\": container with ID starting with 4cdc159f3cda82af18fad00d3f0759f5eca55c1d05791a062104db0281636041 not found: ID does not exist"
Sep 18 19:57:07.782659 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:07.782623    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" path="/var/lib/kubelet/pods/4863ecbc-4a7d-4123-924a-c720779cd0ee/volumes"
Sep 18 19:57:08.937925 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.937889    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938225    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="storage-initializer"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938237    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="storage-initializer"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938253    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kube-rbac-proxy"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kube-rbac-proxy"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938268    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kserve-container"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938273    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kserve-container"
Sep 18 19:57:08.938322 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938316    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kube-rbac-proxy"
Sep 18 19:57:08.938551 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.938328    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4863ecbc-4a7d-4123-924a-c720779cd0ee" containerName="kserve-container"
Sep 18 19:57:08.941842 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.941819    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:08.944175 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.944153    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:57:08.944324 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.944153    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 18 19:57:08.944398 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.944330    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 18 19:57:08.944599 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.944577    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:57:08.945111 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.945091    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:57:08.945247 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.945176    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:57:08.945314 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.945292    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 19:57:08.949859 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:08.949840    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:57:09.046508 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.046469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.046508 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.046515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.046722 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.046541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.046722 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.046601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.046722 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.046677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.147632 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.147594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.147811 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.147639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.147811 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.147664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.147930 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.147828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.147984 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.147928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.148083 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.148061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.148391 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.148373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.148460 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.148420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.150461 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.150444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.156017 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.155994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.253730 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.253645    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:09.524658 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.524575    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:57:09.527123 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:57:09.527097    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63831b49_d18a_4b12_8d44_b4f5523af9a4.slice/crio-a53590004a600dfc49c1693f0bc622f78d818b398b9747f636deea4d5ad555a6 WatchSource:0}: Error finding container a53590004a600dfc49c1693f0bc622f78d818b398b9747f636deea4d5ad555a6: Status 404 returned error can't find the container with id a53590004a600dfc49c1693f0bc622f78d818b398b9747f636deea4d5ad555a6
Sep 18 19:57:09.951545 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.951508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerStarted","Data":"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"}
Sep 18 19:57:09.951943 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:09.951569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerStarted","Data":"a53590004a600dfc49c1693f0bc622f78d818b398b9747f636deea4d5ad555a6"}
Sep 18 19:57:16.984732 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:16.984636    2573 generic.go:356] "Generic (PLEG): container finished" podID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerID="a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895" exitCode=0
Sep 18 19:57:16.984732 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:16.984698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerDied","Data":"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"}
Sep 18 19:57:17.992338 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:17.992297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerStarted","Data":"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"}
Sep 18 19:57:17.992831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:17.992374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerStarted","Data":"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"}
Sep 18 19:57:17.992831 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:17.992604    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:18.011325 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:18.011267    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" podStartSLOduration=2.90625877 podStartE2EDuration="10.011250577s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.104991807s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:18.010115438 +0000 UTC m=+1146.528085083" watchObservedRunningTime="2026-09-18 19:57:18.011250577 +0000 UTC m=+1146.529220217"
Sep 18 19:57:18.996701 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:18.996672    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:57:25.006769 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:57:25.006735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:58:31.357362 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.357322    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-c86222f6"]
Sep 18 19:58:31.360507 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.360483    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:31.365999 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.365959    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-c86222f6"]
Sep 18 19:58:31.451566 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.451533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dj29\" (UniqueName: \"kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29\") pod \"curl-metrics-c86222f6\" (UID: \"f5752cef-efe6-4c24-ac6f-f436bc220831\") " pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:31.552182 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.552146    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dj29\" (UniqueName: \"kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29\") pod \"curl-metrics-c86222f6\" (UID: \"f5752cef-efe6-4c24-ac6f-f436bc220831\") " pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:31.559780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.559754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dj29\" (UniqueName: \"kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29\") pod \"curl-metrics-c86222f6\" (UID: \"f5752cef-efe6-4c24-ac6f-f436bc220831\") " pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:31.678336 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.678250    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:31.943916 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.943843    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-c86222f6"]
Sep 18 19:58:31.947424 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:58:31.947397    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5752cef_efe6_4c24_ac6f_f436bc220831.slice/crio-0dc3efa05f2bbf673ba485ac16e2e1d92871a365bfae0ba0c5df20b8ce5186d5 WatchSource:0}: Error finding container 0dc3efa05f2bbf673ba485ac16e2e1d92871a365bfae0ba0c5df20b8ce5186d5: Status 404 returned error can't find the container with id 0dc3efa05f2bbf673ba485ac16e2e1d92871a365bfae0ba0c5df20b8ce5186d5
Sep 18 19:58:31.949330 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:31.949312    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:58:32.300513 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:32.300425    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-c86222f6" event={"ID":"f5752cef-efe6-4c24-ac6f-f436bc220831","Type":"ContainerStarted","Data":"0dc3efa05f2bbf673ba485ac16e2e1d92871a365bfae0ba0c5df20b8ce5186d5"}
Sep 18 19:58:33.305537 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:33.305500    2573 generic.go:356] "Generic (PLEG): container finished" podID="f5752cef-efe6-4c24-ac6f-f436bc220831" containerID="12387ad03218ffe88d3bd56d9faed185aad4fd7b850b9a9bf1df19004b5b9699" exitCode=0
Sep 18 19:58:33.305938 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:33.305561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-c86222f6" event={"ID":"f5752cef-efe6-4c24-ac6f-f436bc220831","Type":"ContainerDied","Data":"12387ad03218ffe88d3bd56d9faed185aad4fd7b850b9a9bf1df19004b5b9699"}
Sep 18 19:58:34.430343 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.430321    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:34.461402 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.461379    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-c86222f6_f5752cef-efe6-4c24-ac6f-f436bc220831/curl-metrics-c86222f6/0.log"
Sep 18 19:58:34.482787 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.482749    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-c86222f6"]
Sep 18 19:58:34.484022 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.484002    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-c86222f6"]
Sep 18 19:58:34.582072 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.581989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dj29\" (UniqueName: \"kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29\") pod \"f5752cef-efe6-4c24-ac6f-f436bc220831\" (UID: \"f5752cef-efe6-4c24-ac6f-f436bc220831\") "
Sep 18 19:58:34.584076 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.584051    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29" (OuterVolumeSpecName: "kube-api-access-9dj29") pod "f5752cef-efe6-4c24-ac6f-f436bc220831" (UID: "f5752cef-efe6-4c24-ac6f-f436bc220831"). InnerVolumeSpecName "kube-api-access-9dj29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:34.682602 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:34.682570    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dj29\" (UniqueName: \"kubernetes.io/projected/f5752cef-efe6-4c24-ac6f-f436bc220831-kube-api-access-9dj29\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:58:35.314873 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:35.314846    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dc3efa05f2bbf673ba485ac16e2e1d92871a365bfae0ba0c5df20b8ce5186d5"
Sep 18 19:58:35.314873 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:35.314858    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:35.324722 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:58:35.324692    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-c86222f6\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="f5752cef-efe6-4c24-ac6f-f436bc220831" pod="kserve/curl-metrics-c86222f6"
Sep 18 19:58:35.782249 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:35.782215    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5752cef-efe6-4c24-ac6f-f436bc220831" path="/var/lib/kubelet/pods/f5752cef-efe6-4c24-ac6f-f436bc220831/volumes"
Sep 18 19:58:39.854026 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.853992    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-b05caf21"]
Sep 18 19:58:39.854523 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.854314    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5752cef-efe6-4c24-ac6f-f436bc220831" containerName="curl-metrics-c86222f6"
Sep 18 19:58:39.854523 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.854326    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5752cef-efe6-4c24-ac6f-f436bc220831" containerName="curl-metrics-c86222f6"
Sep 18 19:58:39.854523 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.854408    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5752cef-efe6-4c24-ac6f-f436bc220831" containerName="curl-metrics-c86222f6"
Sep 18 19:58:39.857488 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.857471    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:39.862877 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:39.862849    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b05caf21"]
Sep 18 19:58:40.031717 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:40.031680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9k99\" (UniqueName: \"kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99\") pod \"curl-metrics-b05caf21\" (UID: \"214374bd-c514-451d-bc47-1fe39f7aa4b9\") " pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:40.132534 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:40.132440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9k99\" (UniqueName: \"kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99\") pod \"curl-metrics-b05caf21\" (UID: \"214374bd-c514-451d-bc47-1fe39f7aa4b9\") " pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:40.140382 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:40.140360    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9k99\" (UniqueName: \"kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99\") pod \"curl-metrics-b05caf21\" (UID: \"214374bd-c514-451d-bc47-1fe39f7aa4b9\") " pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:40.178081 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:40.178048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:40.449505 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:40.449315    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-b05caf21"]
Sep 18 19:58:40.452358 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:58:40.452328    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod214374bd_c514_451d_bc47_1fe39f7aa4b9.slice/crio-5011d7677c4d9044d2e72b2278d00a2355365295097a9a08909e54dc938a9ed7 WatchSource:0}: Error finding container 5011d7677c4d9044d2e72b2278d00a2355365295097a9a08909e54dc938a9ed7: Status 404 returned error can't find the container with id 5011d7677c4d9044d2e72b2278d00a2355365295097a9a08909e54dc938a9ed7
Sep 18 19:58:41.338942 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:41.338911    2573 generic.go:356] "Generic (PLEG): container finished" podID="214374bd-c514-451d-bc47-1fe39f7aa4b9" containerID="35512b8595276237f866762161f1f19d02bc5a4c6478863d25e6afd7f1ce3728" exitCode=0
Sep 18 19:58:41.339315 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:41.338957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b05caf21" event={"ID":"214374bd-c514-451d-bc47-1fe39f7aa4b9","Type":"ContainerDied","Data":"35512b8595276237f866762161f1f19d02bc5a4c6478863d25e6afd7f1ce3728"}
Sep 18 19:58:41.339315 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:41.338984    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-b05caf21" event={"ID":"214374bd-c514-451d-bc47-1fe39f7aa4b9","Type":"ContainerStarted","Data":"5011d7677c4d9044d2e72b2278d00a2355365295097a9a08909e54dc938a9ed7"}
Sep 18 19:58:42.460617 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.460593    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:42.478981 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.478951    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-b05caf21_214374bd-c514-451d-bc47-1fe39f7aa4b9/curl-metrics-b05caf21/0.log"
Sep 18 19:58:42.501433 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.501405    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-b05caf21"]
Sep 18 19:58:42.505059 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.505036    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-b05caf21"]
Sep 18 19:58:42.550657 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.550634    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9k99\" (UniqueName: \"kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99\") pod \"214374bd-c514-451d-bc47-1fe39f7aa4b9\" (UID: \"214374bd-c514-451d-bc47-1fe39f7aa4b9\") "
Sep 18 19:58:42.552868 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.552844    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99" (OuterVolumeSpecName: "kube-api-access-l9k99") pod "214374bd-c514-451d-bc47-1fe39f7aa4b9" (UID: "214374bd-c514-451d-bc47-1fe39f7aa4b9"). InnerVolumeSpecName "kube-api-access-l9k99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:42.651245 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:42.651134    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9k99\" (UniqueName: \"kubernetes.io/projected/214374bd-c514-451d-bc47-1fe39f7aa4b9-kube-api-access-l9k99\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:58:43.353583 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:43.353550    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5011d7677c4d9044d2e72b2278d00a2355365295097a9a08909e54dc938a9ed7"
Sep 18 19:58:43.353583 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:43.353564    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:43.363707 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:58:43.363678    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-b05caf21\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:43.781566 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:58:43.781477    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-b05caf21\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" pod="kserve/curl-metrics-b05caf21"
Sep 18 19:58:43.783193 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:43.783173    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" path="/var/lib/kubelet/pods/214374bd-c514-451d-bc47-1fe39f7aa4b9/volumes"
Sep 18 19:58:47.859053 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.859018    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 19:58:47.859474 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.859365    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" containerName="curl-metrics-b05caf21"
Sep 18 19:58:47.859474 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.859377    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" containerName="curl-metrics-b05caf21"
Sep 18 19:58:47.859474 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.859426    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="214374bd-c514-451d-bc47-1fe39f7aa4b9" containerName="curl-metrics-b05caf21"
Sep 18 19:58:47.862586 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.862567    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:47.864771 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.864750    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 19:58:47.864771 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.864768    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 18 19:58:47.871500 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.871471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 19:58:47.995744 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.995707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:47.995933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.995759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:47.995933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.995784    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:47.995933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.995841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:47.995933 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:47.995871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jkl9\" (UniqueName: \"kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096528 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096528 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jkl9\" (UniqueName: \"kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.096802 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:58:48.096758    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 19:58:48.097019 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:58:48.096820    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls podName:fe612787-e467-43a3-813b-ec497eb77cd2 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:48.596801742 +0000 UTC m=+1237.114771360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls") pod "isvc-sklearn-predictor-c6b88b959-rbmhx" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 18 19:58:48.097019 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.096987    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.097418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.097394    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.097418 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.097410    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.104961 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.104937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jkl9\" (UniqueName: \"kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.601691 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.601654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.604107 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.604081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-rbmhx\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:48.773306 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:48.773265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:49.042918 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:49.042836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 19:58:49.046387 ip-10-0-136-207 kubenswrapper[2573]: W0918 19:58:49.046346    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe612787_e467_43a3_813b_ec497eb77cd2.slice/crio-ccd127d8b318b680b45f2411fbda5516954ff3dd2c8ea13d066b392c65cd0fc3 WatchSource:0}: Error finding container ccd127d8b318b680b45f2411fbda5516954ff3dd2c8ea13d066b392c65cd0fc3: Status 404 returned error can't find the container with id ccd127d8b318b680b45f2411fbda5516954ff3dd2c8ea13d066b392c65cd0fc3
Sep 18 19:58:49.382804 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:49.382765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerStarted","Data":"3b2d1f50b24f4c66093013d0665429f6e5490930ff55a67076bf9390e2480384"}
Sep 18 19:58:49.382987 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:49.382827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerStarted","Data":"ccd127d8b318b680b45f2411fbda5516954ff3dd2c8ea13d066b392c65cd0fc3"}
Sep 18 19:58:51.812906 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:51.812871    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:58:53.401112 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:53.401074    2573 generic.go:356] "Generic (PLEG): container finished" podID="fe612787-e467-43a3-813b-ec497eb77cd2" containerID="3b2d1f50b24f4c66093013d0665429f6e5490930ff55a67076bf9390e2480384" exitCode=0
Sep 18 19:58:53.401631 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:53.401130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerDied","Data":"3b2d1f50b24f4c66093013d0665429f6e5490930ff55a67076bf9390e2480384"}
Sep 18 19:58:54.408686 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:54.408647    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerStarted","Data":"37eacf7323196c4719ecadfafeb220ef5fccb69d8b07a1737b1b0b131253c9f2"}
Sep 18 19:58:54.409106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:54.408716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerStarted","Data":"e5a31239d91c4799814998e9b43e0a185a38eb210b3096a6fcd952c835f0a998"}
Sep 18 19:58:54.409106 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:54.408910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:54.427224 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:54.427157    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podStartSLOduration=3.620263418 podStartE2EDuration="7.427141965s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806878547s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:54.426260561 +0000 UTC m=+1242.944230205" watchObservedRunningTime="2026-09-18 19:58:54.427141965 +0000 UTC m=+1242.945111604"
Sep 18 19:58:55.413007 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:55.412927    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:58:55.414149 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:55.414123    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:58:56.416786 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:58:56.416744    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:01.421478 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:01.421446    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 19:59:01.422062 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:01.422030    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:02.820388 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:02.820336    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:59:05.602600 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:05.602568    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:59:05.604957 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:05.602803    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kserve-container" containerID="cri-o://7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc" gracePeriod=30
Sep 18 19:59:05.604957 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:05.602863    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kube-rbac-proxy" containerID="cri-o://d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b" gracePeriod=30
Sep 18 19:59:06.464310 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:06.464276    2573 generic.go:356] "Generic (PLEG): container finished" podID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerID="d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b" exitCode=2
Sep 18 19:59:06.464485 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:06.464330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerDied","Data":"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"}
Sep 18 19:59:07.151155 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.151131    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:59:07.260116 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260012    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls\") pod \"63831b49-d18a-4b12-8d44-b4f5523af9a4\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") "
Sep 18 19:59:07.260116 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260074    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location\") pod \"63831b49-d18a-4b12-8d44-b4f5523af9a4\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") "
Sep 18 19:59:07.260116 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260104    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert\") pod \"63831b49-d18a-4b12-8d44-b4f5523af9a4\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") "
Sep 18 19:59:07.260448 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260229    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx\") pod \"63831b49-d18a-4b12-8d44-b4f5523af9a4\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") "
Sep 18 19:59:07.260448 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260277    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config\") pod \"63831b49-d18a-4b12-8d44-b4f5523af9a4\" (UID: \"63831b49-d18a-4b12-8d44-b4f5523af9a4\") "
Sep 18 19:59:07.260558 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260466    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63831b49-d18a-4b12-8d44-b4f5523af9a4" (UID: "63831b49-d18a-4b12-8d44-b4f5523af9a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:07.260712 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260682    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63831b49-d18a-4b12-8d44-b4f5523af9a4" (UID: "63831b49-d18a-4b12-8d44-b4f5523af9a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:07.260802 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.260748    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.262358 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.262334    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63831b49-d18a-4b12-8d44-b4f5523af9a4" (UID: "63831b49-d18a-4b12-8d44-b4f5523af9a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:07.262487 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.262356    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx" (OuterVolumeSpecName: "kube-api-access-crjgx") pod "63831b49-d18a-4b12-8d44-b4f5523af9a4" (UID: "63831b49-d18a-4b12-8d44-b4f5523af9a4"). InnerVolumeSpecName "kube-api-access-crjgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:07.264366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.264348    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63831b49-d18a-4b12-8d44-b4f5523af9a4" (UID: "63831b49-d18a-4b12-8d44-b4f5523af9a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:07.361760 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.361720    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63831b49-d18a-4b12-8d44-b4f5523af9a4-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.361760 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.361751    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63831b49-d18a-4b12-8d44-b4f5523af9a4-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.361760 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.361764    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjgx\" (UniqueName: \"kubernetes.io/projected/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-api-access-crjgx\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.362001 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.361774    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63831b49-d18a-4b12-8d44-b4f5523af9a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 19:59:07.470637 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.470601    2573 generic.go:356] "Generic (PLEG): container finished" podID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerID="7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc" exitCode=0
Sep 18 19:59:07.470797 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.470661    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"
Sep 18 19:59:07.470797 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.470687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerDied","Data":"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"}
Sep 18 19:59:07.470797 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.470755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz" event={"ID":"63831b49-d18a-4b12-8d44-b4f5523af9a4","Type":"ContainerDied","Data":"a53590004a600dfc49c1693f0bc622f78d818b398b9747f636deea4d5ad555a6"}
Sep 18 19:59:07.470797 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.470782    2573 scope.go:117] "RemoveContainer" containerID="d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"
Sep 18 19:59:07.479473 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.479457    2573 scope.go:117] "RemoveContainer" containerID="7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"
Sep 18 19:59:07.486987 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.486971    2573 scope.go:117] "RemoveContainer" containerID="a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"
Sep 18 19:59:07.492112 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.492092    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:59:07.494946 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.494929    2573 scope.go:117] "RemoveContainer" containerID="d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"
Sep 18 19:59:07.495224 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:59:07.495188    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b\": container with ID starting with d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b not found: ID does not exist" containerID="d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"
Sep 18 19:59:07.495279 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495232    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b"} err="failed to get container status \"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b\": rpc error: code = NotFound desc = could not find container \"d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b\": container with ID starting with d093cef53b5e9936f24caccfa577b547394b0f749746a8bd0e1d652cfe69f35b not found: ID does not exist"
Sep 18 19:59:07.495279 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495249    2573 scope.go:117] "RemoveContainer" containerID="7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"
Sep 18 19:59:07.495366 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495287    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-gg7tz"]
Sep 18 19:59:07.495478 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:59:07.495463    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc\": container with ID starting with 7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc not found: ID does not exist" containerID="7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"
Sep 18 19:59:07.495524 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495481    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc"} err="failed to get container status \"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc\": rpc error: code = NotFound desc = could not find container \"7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc\": container with ID starting with 7f2d123d8819dedcdaf8380afcfcbf0ddcb1a17f3b0468a5852b7fe2c559e3fc not found: ID does not exist"
Sep 18 19:59:07.495524 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495494    2573 scope.go:117] "RemoveContainer" containerID="a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"
Sep 18 19:59:07.495726 ip-10-0-136-207 kubenswrapper[2573]: E0918 19:59:07.495708    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895\": container with ID starting with a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895 not found: ID does not exist" containerID="a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"
Sep 18 19:59:07.495780 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.495736    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895"} err="failed to get container status \"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895\": rpc error: code = NotFound desc = could not find container \"a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895\": container with ID starting with a3fd42664de1eaadd730eec0ea8cdaf89d385123e9055e3d2b446198815b0895 not found: ID does not exist"
Sep 18 19:59:07.782304 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:07.782272    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" path="/var/lib/kubelet/pods/63831b49-d18a-4b12-8d44-b4f5523af9a4/volumes"
Sep 18 19:59:11.422377 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:11.422340    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:21.422936 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:21.422896    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:31.421950 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:31.421906    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:41.421950 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:41.421912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:59:51.422481 ip-10-0-136-207 kubenswrapper[2573]: I0918 19:59:51.422435    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:00:01.422363 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:01.422320    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:00:10.778339 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:10.778312    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 20:00:17.936183 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:17.936147    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 20:00:17.936619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:17.936462    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" containerID="cri-o://e5a31239d91c4799814998e9b43e0a185a38eb210b3096a6fcd952c835f0a998" gracePeriod=30
Sep 18 20:00:17.936619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:17.936499    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kube-rbac-proxy" containerID="cri-o://37eacf7323196c4719ecadfafeb220ef5fccb69d8b07a1737b1b0b131253c9f2" gracePeriod=30
Sep 18 20:00:18.767398 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:18.767362    2573 generic.go:356] "Generic (PLEG): container finished" podID="fe612787-e467-43a3-813b-ec497eb77cd2" containerID="37eacf7323196c4719ecadfafeb220ef5fccb69d8b07a1737b1b0b131253c9f2" exitCode=2
Sep 18 20:00:18.767595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:18.767436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerDied","Data":"37eacf7323196c4719ecadfafeb220ef5fccb69d8b07a1737b1b0b131253c9f2"}
Sep 18 20:00:20.778431 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:20.778396    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:00:21.417992 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:21.417943    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:00:22.789494 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:22.789462    2573 generic.go:356] "Generic (PLEG): container finished" podID="fe612787-e467-43a3-813b-ec497eb77cd2" containerID="e5a31239d91c4799814998e9b43e0a185a38eb210b3096a6fcd952c835f0a998" exitCode=0
Sep 18 20:00:22.789851 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:22.789515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerDied","Data":"e5a31239d91c4799814998e9b43e0a185a38eb210b3096a6fcd952c835f0a998"}
Sep 18 20:00:22.879398 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:22.879372    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 20:00:23.010367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010286    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config\") pod \"fe612787-e467-43a3-813b-ec497eb77cd2\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") "
Sep 18 20:00:23.010367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010323    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location\") pod \"fe612787-e467-43a3-813b-ec497eb77cd2\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") "
Sep 18 20:00:23.010367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010351    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert\") pod \"fe612787-e467-43a3-813b-ec497eb77cd2\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") "
Sep 18 20:00:23.010665 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010396    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jkl9\" (UniqueName: \"kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9\") pod \"fe612787-e467-43a3-813b-ec497eb77cd2\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") "
Sep 18 20:00:23.010665 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") pod \"fe612787-e467-43a3-813b-ec497eb77cd2\" (UID: \"fe612787-e467-43a3-813b-ec497eb77cd2\") "
Sep 18 20:00:23.010762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010687    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe612787-e467-43a3-813b-ec497eb77cd2" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:23.010762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010712    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe612787-e467-43a3-813b-ec497eb77cd2" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:23.010762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.010734    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe612787-e467-43a3-813b-ec497eb77cd2" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:23.012580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.012556    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe612787-e467-43a3-813b-ec497eb77cd2" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:23.012707 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.012670    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9" (OuterVolumeSpecName: "kube-api-access-5jkl9") pod "fe612787-e467-43a3-813b-ec497eb77cd2" (UID: "fe612787-e467-43a3-813b-ec497eb77cd2"). InnerVolumeSpecName "kube-api-access-5jkl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:23.111327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.111291    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:00:23.111327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.111321    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe612787-e467-43a3-813b-ec497eb77cd2-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:00:23.111327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.111332    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe612787-e467-43a3-813b-ec497eb77cd2-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:00:23.111558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.111342    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jkl9\" (UniqueName: \"kubernetes.io/projected/fe612787-e467-43a3-813b-ec497eb77cd2-kube-api-access-5jkl9\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:00:23.111558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.111352    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe612787-e467-43a3-813b-ec497eb77cd2-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:00:23.134255 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134220    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:00:23.134540 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134528    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134542    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134553    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134559    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134569    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="storage-initializer"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134575    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="storage-initializer"
Sep 18 20:00:23.134584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134583    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kserve-container"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134588    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kserve-container"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134598    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="storage-initializer"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134603    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="storage-initializer"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134615    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134621    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134668    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kserve-container"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134676    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134681    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" containerName="kube-rbac-proxy"
Sep 18 20:00:23.134769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.134690    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63831b49-d18a-4b12-8d44-b4f5523af9a4" containerName="kserve-container"
Sep 18 20:00:23.136968 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.136951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.139327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.139307    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 18 20:00:23.139495 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.139309    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:00:23.148934 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.148913    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:00:23.313046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.312949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.313046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.312996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzs6\" (UniqueName: \"kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.313284 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.313057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.313284 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.313080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.313284 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.313142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414306 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414306 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzs6\" (UniqueName: \"kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.414854 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.414833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.415058 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.415038    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.415123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.415054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.416821 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.416804    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.422560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.422542    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzs6\" (UniqueName: \"kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.448062 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.448034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:00:23.717287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.717261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:00:23.719576 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:00:23.719549    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a821da7_0845_48db_9235_a72a4998f71d.slice/crio-43e73a6ded91abf654ae619c97e8076f84b16cc7778c3120f17009890c022a62 WatchSource:0}: Error finding container 43e73a6ded91abf654ae619c97e8076f84b16cc7778c3120f17009890c022a62: Status 404 returned error can't find the container with id 43e73a6ded91abf654ae619c97e8076f84b16cc7778c3120f17009890c022a62
Sep 18 20:00:23.796212 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.796169    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"
Sep 18 20:00:23.796212 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.796183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx" event={"ID":"fe612787-e467-43a3-813b-ec497eb77cd2","Type":"ContainerDied","Data":"ccd127d8b318b680b45f2411fbda5516954ff3dd2c8ea13d066b392c65cd0fc3"}
Sep 18 20:00:23.796634 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.796274    2573 scope.go:117] "RemoveContainer" containerID="37eacf7323196c4719ecadfafeb220ef5fccb69d8b07a1737b1b0b131253c9f2"
Sep 18 20:00:23.798049 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.798026    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerStarted","Data":"43e73a6ded91abf654ae619c97e8076f84b16cc7778c3120f17009890c022a62"}
Sep 18 20:00:23.805280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.805261    2573 scope.go:117] "RemoveContainer" containerID="e5a31239d91c4799814998e9b43e0a185a38eb210b3096a6fcd952c835f0a998"
Sep 18 20:00:23.812151 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.812130    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 20:00:23.813875 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.813856    2573 scope.go:117] "RemoveContainer" containerID="3b2d1f50b24f4c66093013d0665429f6e5490930ff55a67076bf9390e2480384"
Sep 18 20:00:23.817570 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:23.817552    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-rbmhx"]
Sep 18 20:00:24.805096 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:24.805063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerStarted","Data":"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"}
Sep 18 20:00:25.782715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:25.782642    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe612787-e467-43a3-813b-ec497eb77cd2" path="/var/lib/kubelet/pods/fe612787-e467-43a3-813b-ec497eb77cd2/volumes"
Sep 18 20:00:27.818479 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:27.818445    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a821da7-0845-48db-9235-a72a4998f71d" containerID="6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb" exitCode=0
Sep 18 20:00:27.818874 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:00:27.818520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerDied","Data":"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"}
Sep 18 20:02:38.471810 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:38.471731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerStarted","Data":"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"}
Sep 18 20:02:38.471810 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:38.471786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerStarted","Data":"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"}
Sep 18 20:02:38.472257 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:38.471895    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:02:38.472257 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:38.471957    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:02:38.490606 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:38.490561    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podStartSLOduration=1.205050809 podStartE2EDuration="2m15.490545624s" totalImagesPullingTime="2m10.351750001s" totalInitContainerRuntime="3.933744814s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:38.48894384 +0000 UTC m=+1467.006913479" watchObservedRunningTime="2026-09-18 20:02:38.490545624 +0000 UTC m=+1467.008515325"
Sep 18 20:02:44.481391 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:02:44.481363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:03:14.526741 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:14.526695    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 20:03:24.483767 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:24.483731    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:03:33.244147 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:33.244068    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:03:33.244724 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:33.244379    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container" containerID="cri-o://add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba" gracePeriod=30
Sep 18 20:03:33.244724 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:33.244399    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy" containerID="cri-o://7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf" gracePeriod=30
Sep 18 20:03:33.699476 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:33.699447    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a821da7-0845-48db-9235-a72a4998f71d" containerID="7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf" exitCode=2
Sep 18 20:03:33.699633 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:33.699516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerDied","Data":"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"}
Sep 18 20:03:34.476000 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:34.475961    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 20:03:35.523388 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:35.523343    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:03:38.458754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.458722    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:03:38.461279 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.461259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.463898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.463870    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 18 20:03:38.464003 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.463917    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:03:38.472262 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.472241    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:03:38.588157 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.588122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.588338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.588174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.588338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.588220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.588338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.588271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.588338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.588287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7h9\" (UniqueName: \"kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688768 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.688734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688768 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.688766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7h9\" (UniqueName: \"kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.688797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.688831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.688849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.688994 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:03:38.688896    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 20:03:38.688994 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:03:38.688979    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls podName:5a97ce8f-6a86-434e-aae8-31c910d03615 nodeName:}" failed. No retries permitted until 2026-09-18 20:03:39.188958746 +0000 UTC m=+1527.706928381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls") pod "isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 18 20:03:38.689307 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.689291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.689518 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.689502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.689558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.689529    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:38.697021 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:38.697000    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7h9\" (UniqueName: \"kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:39.193566 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.193528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:39.195947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.195923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:39.371578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.371541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:39.475785 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.475708    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 20:03:39.641083 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.641047    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:03:39.643414 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:03:39.643389    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a97ce8f_6a86_434e_aae8_31c910d03615.slice/crio-b80649b76a644d7203a77e3ca81b064f0a58a68f7a78ad6f14c5d5ff71c0ae77 WatchSource:0}: Error finding container b80649b76a644d7203a77e3ca81b064f0a58a68f7a78ad6f14c5d5ff71c0ae77: Status 404 returned error can't find the container with id b80649b76a644d7203a77e3ca81b064f0a58a68f7a78ad6f14c5d5ff71c0ae77
Sep 18 20:03:39.726101 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.726033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerStarted","Data":"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"}
Sep 18 20:03:39.726233 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:39.726100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerStarted","Data":"b80649b76a644d7203a77e3ca81b064f0a58a68f7a78ad6f14c5d5ff71c0ae77"}
Sep 18 20:03:40.480045 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.480017    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:03:40.607355 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607327    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhzs6\" (UniqueName: \"kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6\") pod \"7a821da7-0845-48db-9235-a72a4998f71d\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") "
Sep 18 20:03:40.607561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607369    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert\") pod \"7a821da7-0845-48db-9235-a72a4998f71d\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") "
Sep 18 20:03:40.607561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607442    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config\") pod \"7a821da7-0845-48db-9235-a72a4998f71d\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") "
Sep 18 20:03:40.607561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location\") pod \"7a821da7-0845-48db-9235-a72a4998f71d\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") "
Sep 18 20:03:40.607561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607521    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls\") pod \"7a821da7-0845-48db-9235-a72a4998f71d\" (UID: \"7a821da7-0845-48db-9235-a72a4998f71d\") "
Sep 18 20:03:40.607801 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607774    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a821da7-0845-48db-9235-a72a4998f71d" (UID: "7a821da7-0845-48db-9235-a72a4998f71d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:40.607854 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607792    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a821da7-0845-48db-9235-a72a4998f71d" (UID: "7a821da7-0845-48db-9235-a72a4998f71d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:40.607854 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.607785    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a821da7-0845-48db-9235-a72a4998f71d" (UID: "7a821da7-0845-48db-9235-a72a4998f71d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:40.609569 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.609542    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a821da7-0845-48db-9235-a72a4998f71d" (UID: "7a821da7-0845-48db-9235-a72a4998f71d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:40.609652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.609635    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6" (OuterVolumeSpecName: "kube-api-access-mhzs6") pod "7a821da7-0845-48db-9235-a72a4998f71d" (UID: "7a821da7-0845-48db-9235-a72a4998f71d"). InnerVolumeSpecName "kube-api-access-mhzs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:40.708375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.708345    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:03:40.708375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.708372    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a821da7-0845-48db-9235-a72a4998f71d-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:03:40.708375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.708381    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a821da7-0845-48db-9235-a72a4998f71d-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:03:40.708593 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.708391    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzs6\" (UniqueName: \"kubernetes.io/projected/7a821da7-0845-48db-9235-a72a4998f71d-kube-api-access-mhzs6\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:03:40.708593 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.708399    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a821da7-0845-48db-9235-a72a4998f71d-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:03:40.731636 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.731605    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a821da7-0845-48db-9235-a72a4998f71d" containerID="add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba" exitCode=0
Sep 18 20:03:40.731770 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.731671    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"
Sep 18 20:03:40.731770 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.731693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerDied","Data":"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"}
Sep 18 20:03:40.731770 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.731753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6" event={"ID":"7a821da7-0845-48db-9235-a72a4998f71d","Type":"ContainerDied","Data":"43e73a6ded91abf654ae619c97e8076f84b16cc7778c3120f17009890c022a62"}
Sep 18 20:03:40.731893 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.731781    2573 scope.go:117] "RemoveContainer" containerID="7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"
Sep 18 20:03:40.739913 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.739892    2573 scope.go:117] "RemoveContainer" containerID="add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"
Sep 18 20:03:40.746963 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.746947    2573 scope.go:117] "RemoveContainer" containerID="6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"
Sep 18 20:03:40.751644 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.751624    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:03:40.754569 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.754552    2573 scope.go:117] "RemoveContainer" containerID="7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"
Sep 18 20:03:40.754865 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:03:40.754846    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf\": container with ID starting with 7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf not found: ID does not exist" containerID="7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"
Sep 18 20:03:40.754915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.754874    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf"} err="failed to get container status \"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf\": rpc error: code = NotFound desc = could not find container \"7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf\": container with ID starting with 7a52620133bcb1822fa3a69064b50b23879988bd7df0ca53d9fe365d27015faf not found: ID does not exist"
Sep 18 20:03:40.754915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.754891    2573 scope.go:117] "RemoveContainer" containerID="add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"
Sep 18 20:03:40.755174 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:03:40.755157    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba\": container with ID starting with add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba not found: ID does not exist" containerID="add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"
Sep 18 20:03:40.755247 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.755179    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba"} err="failed to get container status \"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba\": rpc error: code = NotFound desc = could not find container \"add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba\": container with ID starting with add4e2544b8a6dab68ecece5af7ab063d535d6114baa8c45715a2d5ce6a65fba not found: ID does not exist"
Sep 18 20:03:40.755247 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.755192    2573 scope.go:117] "RemoveContainer" containerID="6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"
Sep 18 20:03:40.755468 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:03:40.755447    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb\": container with ID starting with 6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb not found: ID does not exist" containerID="6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"
Sep 18 20:03:40.755537 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.755470    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wsls6"]
Sep 18 20:03:40.755537 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:40.755477    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb"} err="failed to get container status \"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb\": rpc error: code = NotFound desc = could not find container \"6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb\": container with ID starting with 6cf2c32296ed35b51d5fd14b88141964311521b22bd660a027faea9bdaba38bb not found: ID does not exist"
Sep 18 20:03:41.781902 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:41.781870    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a821da7-0845-48db-9235-a72a4998f71d" path="/var/lib/kubelet/pods/7a821da7-0845-48db-9235-a72a4998f71d/volumes"
Sep 18 20:03:45.753041 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:45.753009    2573 generic.go:356] "Generic (PLEG): container finished" podID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerID="810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2" exitCode=0
Sep 18 20:03:45.753530 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:45.753087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerDied","Data":"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"}
Sep 18 20:03:46.759771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:46.759737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerStarted","Data":"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"}
Sep 18 20:03:46.760158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:46.759802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerStarted","Data":"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"}
Sep 18 20:03:46.760158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:46.759992    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:46.778570 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:46.778528    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podStartSLOduration=2.810760077 podStartE2EDuration="8.778499841s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.967739764s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:46.776859772 +0000 UTC m=+1535.294829410" watchObservedRunningTime="2026-09-18 20:03:46.778499841 +0000 UTC m=+1535.296469483"
Sep 18 20:03:47.764847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:47.764812    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:47.766218 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:47.766168    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:03:48.769596 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:48.769548    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:03:53.774529 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:53.774501    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:03:53.774953 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:03:53.774929    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 20:04:03.776343 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:03.776310    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:04:10.404400 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:10.404364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:04:10.405092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:10.404661    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container" containerID="cri-o://017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42" gracePeriod=30
Sep 18 20:04:10.405092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:10.404724    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kube-rbac-proxy" containerID="cri-o://0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00" gracePeriod=30
Sep 18 20:04:10.877146 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:10.877114    2573 generic.go:356] "Generic (PLEG): container finished" podID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerID="0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00" exitCode=2
Sep 18 20:04:10.877332 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:10.877159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerDied","Data":"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"}
Sep 18 20:04:11.639754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.639731    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:04:11.767265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767167    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert\") pod \"5a97ce8f-6a86-434e-aae8-31c910d03615\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") "
Sep 18 20:04:11.767265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767246    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config\") pod \"5a97ce8f-6a86-434e-aae8-31c910d03615\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") "
Sep 18 20:04:11.767463 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767282    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location\") pod \"5a97ce8f-6a86-434e-aae8-31c910d03615\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") "
Sep 18 20:04:11.767463 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767375    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") pod \"5a97ce8f-6a86-434e-aae8-31c910d03615\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") "
Sep 18 20:04:11.767463 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd7h9\" (UniqueName: \"kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9\") pod \"5a97ce8f-6a86-434e-aae8-31c910d03615\" (UID: \"5a97ce8f-6a86-434e-aae8-31c910d03615\") "
Sep 18 20:04:11.767665 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a97ce8f-6a86-434e-aae8-31c910d03615" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:11.767717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.767658    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a97ce8f-6a86-434e-aae8-31c910d03615" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:11.769544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.769511    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a97ce8f-6a86-434e-aae8-31c910d03615" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:11.769544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.769532    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9" (OuterVolumeSpecName: "kube-api-access-wd7h9") pod "5a97ce8f-6a86-434e-aae8-31c910d03615" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615"). InnerVolumeSpecName "kube-api-access-wd7h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:11.798046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.798019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a97ce8f-6a86-434e-aae8-31c910d03615" (UID: "5a97ce8f-6a86-434e-aae8-31c910d03615"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:11.868267 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.868239    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7h9\" (UniqueName: \"kubernetes.io/projected/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-api-access-wd7h9\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:04:11.868267 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.868264    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:04:11.868452 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.868280    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a97ce8f-6a86-434e-aae8-31c910d03615-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:04:11.868452 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.868294    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a97ce8f-6a86-434e-aae8-31c910d03615-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:04:11.868452 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.868306    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a97ce8f-6a86-434e-aae8-31c910d03615-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:04:11.883571 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.883544    2573 generic.go:356] "Generic (PLEG): container finished" podID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerID="017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42" exitCode=0
Sep 18 20:04:11.883692 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.883585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerDied","Data":"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"}
Sep 18 20:04:11.883692 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.883614    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"
Sep 18 20:04:11.883692 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.883632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz" event={"ID":"5a97ce8f-6a86-434e-aae8-31c910d03615","Type":"ContainerDied","Data":"b80649b76a644d7203a77e3ca81b064f0a58a68f7a78ad6f14c5d5ff71c0ae77"}
Sep 18 20:04:11.883692 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.883656    2573 scope.go:117] "RemoveContainer" containerID="0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"
Sep 18 20:04:11.891468 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.891365    2573 scope.go:117] "RemoveContainer" containerID="017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"
Sep 18 20:04:11.898481 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.898466    2573 scope.go:117] "RemoveContainer" containerID="810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"
Sep 18 20:04:11.905512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.905476    2573 scope.go:117] "RemoveContainer" containerID="0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"
Sep 18 20:04:11.906136 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:04:11.905865    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00\": container with ID starting with 0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00 not found: ID does not exist" containerID="0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"
Sep 18 20:04:11.906136 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.905951    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00"} err="failed to get container status \"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00\": rpc error: code = NotFound desc = could not find container \"0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00\": container with ID starting with 0d3b52eb867ea94ecab040a0561a4ff18a9be9b2e3c20cd3aad3fb1f567a8c00 not found: ID does not exist"
Sep 18 20:04:11.906136 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.905974    2573 scope.go:117] "RemoveContainer" containerID="017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"
Sep 18 20:04:11.906611 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:04:11.906485    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42\": container with ID starting with 017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42 not found: ID does not exist" containerID="017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"
Sep 18 20:04:11.906611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.906517    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42"} err="failed to get container status \"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42\": rpc error: code = NotFound desc = could not find container \"017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42\": container with ID starting with 017882d85ed3420a5aa3e7d8a9d4c68b6c61a76432cb7726043691ed901c3e42 not found: ID does not exist"
Sep 18 20:04:11.906611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.906541    2573 scope.go:117] "RemoveContainer" containerID="810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"
Sep 18 20:04:11.907038 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:04:11.907018    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2\": container with ID starting with 810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2 not found: ID does not exist" containerID="810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"
Sep 18 20:04:11.907113 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.907044    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2"} err="failed to get container status \"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2\": rpc error: code = NotFound desc = could not find container \"810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2\": container with ID starting with 810cf3457d29c3b23873640d277ad889d4f01a0e291fdd01de85373f417f16e2 not found: ID does not exist"
Sep 18 20:04:11.908375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.908352    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:04:11.911292 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:11.911273    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-cpcgz"]
Sep 18 20:04:13.782298 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:13.782266    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" path="/var/lib/kubelet/pods/5a97ce8f-6a86-434e-aae8-31c910d03615/volumes"
Sep 18 20:04:15.580573 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580542    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580878    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="storage-initializer"
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580891    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="storage-initializer"
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580902    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container"
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580910    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container"
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580924    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="storage-initializer"
Sep 18 20:04:15.580930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580929    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="storage-initializer"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580937    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580942    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580948    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580955    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580965    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.580970    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.581017    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kserve-container"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.581028    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.581033    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a97ce8f-6a86-434e-aae8-31c910d03615" containerName="kube-rbac-proxy"
Sep 18 20:04:15.581123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.581040    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a821da7-0845-48db-9235-a72a4998f71d" containerName="kserve-container"
Sep 18 20:04:15.584024 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.584006    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.586375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586341    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 18 20:04:15.586375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586365    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:04:15.586560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586350    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:04:15.586560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:04:15.586560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586452    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:04:15.586560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.586373    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:04:15.587150 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.587137    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:04:15.593259 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.593239    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:04:15.702588 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.702549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.702783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.702602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxfs\" (UniqueName: \"kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.702783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.702677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.702783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.702719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.702783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.702744    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.803505 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.803675 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803526    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.803675 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxfs\" (UniqueName: \"kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.803780 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.803780 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.804007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.803988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.804258 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.804238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.804363 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.804343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.806084 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.806069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.810710 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.810690    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxfs\" (UniqueName: \"kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:15.894709 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:15.894676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:16.166661 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:16.166639    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:04:16.169007 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:04:16.168974    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb426e235_1eda_467d_b29b_f7f87127123f.slice/crio-440e7322579e5ecd89927180cf90cf0d730a780d66389cb30ef78437cdc1b45f WatchSource:0}: Error finding container 440e7322579e5ecd89927180cf90cf0d730a780d66389cb30ef78437cdc1b45f: Status 404 returned error can't find the container with id 440e7322579e5ecd89927180cf90cf0d730a780d66389cb30ef78437cdc1b45f
Sep 18 20:04:16.905858 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:16.905818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerStarted","Data":"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"}
Sep 18 20:04:16.906253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:16.905886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerStarted","Data":"440e7322579e5ecd89927180cf90cf0d730a780d66389cb30ef78437cdc1b45f"}
Sep 18 20:04:20.923691 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:20.923657    2573 generic.go:356] "Generic (PLEG): container finished" podID="b426e235-1eda-467d-b29b-f7f87127123f" containerID="e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8" exitCode=0
Sep 18 20:04:20.924079 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:20.923729    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerDied","Data":"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"}
Sep 18 20:04:21.930454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:21.930420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerStarted","Data":"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"}
Sep 18 20:04:21.930839 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:21.930480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerStarted","Data":"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"}
Sep 18 20:04:21.930839 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:21.930686    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:21.930839 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:21.930741    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:21.950542 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:21.950501    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podStartSLOduration=3.069376536 podStartE2EDuration="6.950487716s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88111118s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:21.949116993 +0000 UTC m=+1570.467086633" watchObservedRunningTime="2026-09-18 20:04:21.950487716 +0000 UTC m=+1570.468457356"
Sep 18 20:04:27.939820 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:27.939789    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:04:31.718331 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.718299    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:04:31.721049 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.721032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.723086 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.723061    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 20:04:31.723364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.723347    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:04:31.733464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.733442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:04:31.754272 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.754251    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:04:31.757246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.757232    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.759750 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.759732    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:04:31.777021 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.776999    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:04:31.836238 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.836190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.836367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.836289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.836367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.836322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfrk\" (UniqueName: \"kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.836367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.836344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.836499 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.836392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.936797 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.936770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.936951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.936807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.936951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.936827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphj8\" (UniqueName: \"kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.937073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.936949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.937073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.936998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.937253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfrk\" (UniqueName: \"kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937397 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:31.937432 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.937802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.937783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.939409 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.939388    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:31.945453 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:31.945434    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfrk\" (UniqueName: \"kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk\") pod \"isvc-canary-create-v2-predictor-579784d7d7-fjw2s\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:32.037547 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:32.037782 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.037892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.037892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphj8\" (UniqueName: \"kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.037892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.038051 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.037925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.038354 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.038332    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.038627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.038610    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.038681 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.038648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.040280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.040264    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.046687 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.046662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphj8\" (UniqueName: \"kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8\") pod \"isvc-canary-create-predictor-68984b9498-sf9gn\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.066041 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.066001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:32.309010 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.308931    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:04:32.312283 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:04:32.312256    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f06168_f642_42fc_9eef_687919d50d31.slice/crio-d5d1c5c509cc1e66cfa682b40f6ed8d02f7a962c7fc7a0ad52abea1f3f023769 WatchSource:0}: Error finding container d5d1c5c509cc1e66cfa682b40f6ed8d02f7a962c7fc7a0ad52abea1f3f023769: Status 404 returned error can't find the container with id d5d1c5c509cc1e66cfa682b40f6ed8d02f7a962c7fc7a0ad52abea1f3f023769
Sep 18 20:04:32.344885 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.344867    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:04:32.346681 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:04:32.346655    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ed148d_ef6a_4274_a05d_9c2c508ab3f8.slice/crio-1f664ef3b5cc6c539555a44dca5ce9a762287abaa3d3cba8317a156a6619efdf WatchSource:0}: Error finding container 1f664ef3b5cc6c539555a44dca5ce9a762287abaa3d3cba8317a156a6619efdf: Status 404 returned error can't find the container with id 1f664ef3b5cc6c539555a44dca5ce9a762287abaa3d3cba8317a156a6619efdf
Sep 18 20:04:32.977190 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.977154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerStarted","Data":"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"}
Sep 18 20:04:32.977764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.977242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerStarted","Data":"1f664ef3b5cc6c539555a44dca5ce9a762287abaa3d3cba8317a156a6619efdf"}
Sep 18 20:04:32.978722 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.978696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerStarted","Data":"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"}
Sep 18 20:04:32.978851 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:32.978750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerStarted","Data":"d5d1c5c509cc1e66cfa682b40f6ed8d02f7a962c7fc7a0ad52abea1f3f023769"}
Sep 18 20:04:36.997813 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:36.997781    2573 generic.go:356] "Generic (PLEG): container finished" podID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerID="811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184" exitCode=0
Sep 18 20:04:36.998297 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:36.997854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerDied","Data":"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"}
Sep 18 20:04:36.999454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:36.999431    2573 generic.go:356] "Generic (PLEG): container finished" podID="64f06168-f642-42fc-9eef-687919d50d31" containerID="e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95" exitCode=0
Sep 18 20:04:36.999549 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:36.999489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerDied","Data":"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"}
Sep 18 20:04:38.005862 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.005827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerStarted","Data":"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"}
Sep 18 20:04:38.006298 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.005901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerStarted","Data":"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"}
Sep 18 20:04:38.006298 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.006144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:38.008103 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.008082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerStarted","Data":"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"}
Sep 18 20:04:38.008234 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.008126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerStarted","Data":"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"}
Sep 18 20:04:38.008384 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.008368    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:38.008427 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.008407    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:38.009764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.009741    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:04:38.024595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.024551    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podStartSLOduration=3.134003189 podStartE2EDuration="7.024538388s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.890535199s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:38.022800648 +0000 UTC m=+1586.540770288" watchObservedRunningTime="2026-09-18 20:04:38.024538388 +0000 UTC m=+1586.542508027"
Sep 18 20:04:38.040609 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:38.040572    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podStartSLOduration=3.283727667 podStartE2EDuration="7.040560323s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.756832656s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:38.038711381 +0000 UTC m=+1586.556681022" watchObservedRunningTime="2026-09-18 20:04:38.040560323 +0000 UTC m=+1586.558529962"
Sep 18 20:04:39.013598 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:39.013561    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:04:39.013980 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:39.013687    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:39.015004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:39.014977    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:04:40.017789 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:40.017744    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:04:44.018081 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:44.018049    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:04:44.018662 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:44.018636    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:04:45.022226 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:45.022178    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:04:45.022799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:45.022776    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:04:54.019288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:54.019237    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:04:55.022852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:55.022812    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:04:58.026866 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:04:58.026814    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 20:05:04.019164 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:04.019125    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:05:05.023017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:05.022981    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:05:07.943248 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:07.943196    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:05:14.019356 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:14.019317    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:05:14.709001 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:14.708971    2573 scope.go:117] "RemoveContainer" containerID="35512b8595276237f866762161f1f19d02bc5a4c6478863d25e6afd7f1ce3728"
Sep 18 20:05:14.716884 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:14.716864    2573 scope.go:117] "RemoveContainer" containerID="12387ad03218ffe88d3bd56d9faed185aad4fd7b850b9a9bf1df19004b5b9699"
Sep 18 20:05:15.023052 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:15.022966    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:05:15.633864 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:15.633830    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:05:15.634157 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:15.634111    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container" containerID="cri-o://d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556" gracePeriod=30
Sep 18 20:05:15.634266 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:15.634170    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kube-rbac-proxy" containerID="cri-o://98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae" gracePeriod=30
Sep 18 20:05:16.182145 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:16.182114    2573 generic.go:356] "Generic (PLEG): container finished" podID="b426e235-1eda-467d-b29b-f7f87127123f" containerID="98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae" exitCode=2
Sep 18 20:05:16.182544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:16.182189    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerDied","Data":"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"}
Sep 18 20:05:17.934938 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:17.934896    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 20:05:18.981358 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:18.981318    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:05:22.884778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:22.884756    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:05:23.068976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.068890    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtxfs\" (UniqueName: \"kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs\") pod \"b426e235-1eda-467d-b29b-f7f87127123f\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") "
Sep 18 20:05:23.068976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.068944    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location\") pod \"b426e235-1eda-467d-b29b-f7f87127123f\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") "
Sep 18 20:05:23.069163 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069014    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert\") pod \"b426e235-1eda-467d-b29b-f7f87127123f\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") "
Sep 18 20:05:23.069163 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069109    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls\") pod \"b426e235-1eda-467d-b29b-f7f87127123f\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") "
Sep 18 20:05:23.069163 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069152    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config\") pod \"b426e235-1eda-467d-b29b-f7f87127123f\" (UID: \"b426e235-1eda-467d-b29b-f7f87127123f\") "
Sep 18 20:05:23.069355 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069328    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b426e235-1eda-467d-b29b-f7f87127123f" (UID: "b426e235-1eda-467d-b29b-f7f87127123f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:23.069414 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069369    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b426e235-1eda-467d-b29b-f7f87127123f" (UID: "b426e235-1eda-467d-b29b-f7f87127123f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:23.069545 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069524    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b426e235-1eda-467d-b29b-f7f87127123f" (UID: "b426e235-1eda-467d-b29b-f7f87127123f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:23.069604 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069560    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:05:23.069604 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.069571    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b426e235-1eda-467d-b29b-f7f87127123f-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:05:23.071130 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.071101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs" (OuterVolumeSpecName: "kube-api-access-jtxfs") pod "b426e235-1eda-467d-b29b-f7f87127123f" (UID: "b426e235-1eda-467d-b29b-f7f87127123f"). InnerVolumeSpecName "kube-api-access-jtxfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:23.071130 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.071121    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b426e235-1eda-467d-b29b-f7f87127123f" (UID: "b426e235-1eda-467d-b29b-f7f87127123f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:23.170068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.170040    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtxfs\" (UniqueName: \"kubernetes.io/projected/b426e235-1eda-467d-b29b-f7f87127123f-kube-api-access-jtxfs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:05:23.170068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.170063    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b426e235-1eda-467d-b29b-f7f87127123f-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:05:23.170068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.170074    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b426e235-1eda-467d-b29b-f7f87127123f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:05:23.212473 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.212443    2573 generic.go:356] "Generic (PLEG): container finished" podID="b426e235-1eda-467d-b29b-f7f87127123f" containerID="d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556" exitCode=0
Sep 18 20:05:23.212580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.212500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerDied","Data":"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"}
Sep 18 20:05:23.212580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.212509    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"
Sep 18 20:05:23.212580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.212563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5" event={"ID":"b426e235-1eda-467d-b29b-f7f87127123f","Type":"ContainerDied","Data":"440e7322579e5ecd89927180cf90cf0d730a780d66389cb30ef78437cdc1b45f"}
Sep 18 20:05:23.212683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.212599    2573 scope.go:117] "RemoveContainer" containerID="98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"
Sep 18 20:05:23.220977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.220964    2573 scope.go:117] "RemoveContainer" containerID="d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"
Sep 18 20:05:23.229664 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.229645    2573 scope.go:117] "RemoveContainer" containerID="e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"
Sep 18 20:05:23.233502 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.233477    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:05:23.236561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.236541    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-mqjh5"]
Sep 18 20:05:23.238128 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238109    2573 scope.go:117] "RemoveContainer" containerID="98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"
Sep 18 20:05:23.238432 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:05:23.238413    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae\": container with ID starting with 98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae not found: ID does not exist" containerID="98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"
Sep 18 20:05:23.238504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238439    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae"} err="failed to get container status \"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae\": rpc error: code = NotFound desc = could not find container \"98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae\": container with ID starting with 98e57857f99bce2a95639e917ffec825ff8da9add5b730da1c546e2310711bae not found: ID does not exist"
Sep 18 20:05:23.238504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238455    2573 scope.go:117] "RemoveContainer" containerID="d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"
Sep 18 20:05:23.238698 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:05:23.238681    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556\": container with ID starting with d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556 not found: ID does not exist" containerID="d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"
Sep 18 20:05:23.238744 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238702    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556"} err="failed to get container status \"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556\": rpc error: code = NotFound desc = could not find container \"d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556\": container with ID starting with d85761cfbf12c1cfe43e2057d830d4c05e104e11e9ec89f091804de1083d9556 not found: ID does not exist"
Sep 18 20:05:23.238744 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238716    2573 scope.go:117] "RemoveContainer" containerID="e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"
Sep 18 20:05:23.238908 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:05:23.238893    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8\": container with ID starting with e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8 not found: ID does not exist" containerID="e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"
Sep 18 20:05:23.238947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.238914    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8"} err="failed to get container status \"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8\": rpc error: code = NotFound desc = could not find container \"e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8\": container with ID starting with e3bcc6ad3822916c9d2ad2edae0b9329643ae7443561d38e05c4d557b751adb8 not found: ID does not exist"
Sep 18 20:05:23.787098 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:23.787065    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b426e235-1eda-467d-b29b-f7f87127123f" path="/var/lib/kubelet/pods/b426e235-1eda-467d-b29b-f7f87127123f/volumes"
Sep 18 20:05:24.018771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:24.018730    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:05:25.023747 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:25.023707    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:05:34.019122 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:34.019082    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:05:35.022978 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:35.022943    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:05:44.018859 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:44.018823    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:05:45.023752 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:45.023709    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:05:54.020177 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:54.020149    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:05:55.023350 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:05:55.023318    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:06:01.795323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.795286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:06:01.795711 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.795591    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" containerID="cri-o://b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686" gracePeriod=30
Sep 18 20:06:01.795711 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.795677    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kube-rbac-proxy" containerID="cri-o://979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3" gracePeriod=30
Sep 18 20:06:01.802254 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.802230    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:06:01.802563 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.802514    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" containerID="cri-o://27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309" gracePeriod=30
Sep 18 20:06:01.802672 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:01.802589    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kube-rbac-proxy" containerID="cri-o://89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232" gracePeriod=30
Sep 18 20:06:02.373253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:02.373222    2573 generic.go:356] "Generic (PLEG): container finished" podID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerID="979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3" exitCode=2
Sep 18 20:06:02.373419 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:02.373294    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerDied","Data":"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"}
Sep 18 20:06:02.375236 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:02.375212    2573 generic.go:356] "Generic (PLEG): container finished" podID="64f06168-f642-42fc-9eef-687919d50d31" containerID="89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232" exitCode=2
Sep 18 20:06:02.375336 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:02.375292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerDied","Data":"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"}
Sep 18 20:06:04.014514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:04.014476    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 18 20:06:04.018719 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:04.018694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 20:06:05.018650 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:05.018609    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 20:06:05.022877 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:05.022847    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 20:06:06.532269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.532246    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:06:06.628150 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628126    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert\") pod \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") "
Sep 18 20:06:06.628338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location\") pod \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") "
Sep 18 20:06:06.628338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628252    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls\") pod \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") "
Sep 18 20:06:06.628338 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628308    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dphj8\" (UniqueName: \"kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8\") pod \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") "
Sep 18 20:06:06.628502 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628356    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config\") pod \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\" (UID: \"41ed148d-ef6a-4274-a05d-9c2c508ab3f8\") "
Sep 18 20:06:06.628553 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628520    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41ed148d-ef6a-4274-a05d-9c2c508ab3f8" (UID: "41ed148d-ef6a-4274-a05d-9c2c508ab3f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:06.628606 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628550    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41ed148d-ef6a-4274-a05d-9c2c508ab3f8" (UID: "41ed148d-ef6a-4274-a05d-9c2c508ab3f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:06.628756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628713    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.628756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628736    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.628944 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.628798    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41ed148d-ef6a-4274-a05d-9c2c508ab3f8" (UID: "41ed148d-ef6a-4274-a05d-9c2c508ab3f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:06.630578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.630549    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41ed148d-ef6a-4274-a05d-9c2c508ab3f8" (UID: "41ed148d-ef6a-4274-a05d-9c2c508ab3f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:06.630578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.630556    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8" (OuterVolumeSpecName: "kube-api-access-dphj8") pod "41ed148d-ef6a-4274-a05d-9c2c508ab3f8" (UID: "41ed148d-ef6a-4274-a05d-9c2c508ab3f8"). InnerVolumeSpecName "kube-api-access-dphj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:06.644591 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.644574    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:06:06.729999 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.729977    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location\") pod \"64f06168-f642-42fc-9eef-687919d50d31\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") "
Sep 18 20:06:06.730113 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730008    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert\") pod \"64f06168-f642-42fc-9eef-687919d50d31\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") "
Sep 18 20:06:06.730113 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730070    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls\") pod \"64f06168-f642-42fc-9eef-687919d50d31\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") "
Sep 18 20:06:06.730113 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730085    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config\") pod \"64f06168-f642-42fc-9eef-687919d50d31\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") "
Sep 18 20:06:06.730319 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpfrk\" (UniqueName: \"kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk\") pod \"64f06168-f642-42fc-9eef-687919d50d31\" (UID: \"64f06168-f642-42fc-9eef-687919d50d31\") "
Sep 18 20:06:06.730381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730321    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64f06168-f642-42fc-9eef-687919d50d31" (UID: "64f06168-f642-42fc-9eef-687919d50d31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:06.730471 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphj8\" (UniqueName: \"kubernetes.io/projected/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-api-access-dphj8\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.730532 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730474    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.730532 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730484    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41ed148d-ef6a-4274-a05d-9c2c508ab3f8-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.730532 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64f06168-f642-42fc-9eef-687919d50d31" (UID: "64f06168-f642-42fc-9eef-687919d50d31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:06.730532 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.730499    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64f06168-f642-42fc-9eef-687919d50d31" (UID: "64f06168-f642-42fc-9eef-687919d50d31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:06.732104 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.732085    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64f06168-f642-42fc-9eef-687919d50d31" (UID: "64f06168-f642-42fc-9eef-687919d50d31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:06.732170 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.732115    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk" (OuterVolumeSpecName: "kube-api-access-mpfrk") pod "64f06168-f642-42fc-9eef-687919d50d31" (UID: "64f06168-f642-42fc-9eef-687919d50d31"). InnerVolumeSpecName "kube-api-access-mpfrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:06.831484 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.831457    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfrk\" (UniqueName: \"kubernetes.io/projected/64f06168-f642-42fc-9eef-687919d50d31-kube-api-access-mpfrk\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.831484 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.831480    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f06168-f642-42fc-9eef-687919d50d31-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.831484 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.831490    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.831715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.831500    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64f06168-f642-42fc-9eef-687919d50d31-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.831715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.831509    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64f06168-f642-42fc-9eef-687919d50d31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:06:06.964662 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.964586    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:06:06.965059 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965043    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="storage-initializer"
Sep 18 20:06:06.965110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965063    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="storage-initializer"
Sep 18 20:06:06.965110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965076    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container"
Sep 18 20:06:06.965110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965086    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container"
Sep 18 20:06:06.965110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965103    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965112    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965127    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965136    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965150    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965159    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965171    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="storage-initializer"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965180    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="storage-initializer"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965211    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965220    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965231    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965239    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965256    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="storage-initializer"
Sep 18 20:06:06.965269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965265    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="storage-initializer"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965350    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kserve-container"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965364    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965376    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kserve-container"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965385    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f06168-f642-42fc-9eef-687919d50d31" containerName="kube-rbac-proxy"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965395    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b426e235-1eda-467d-b29b-f7f87127123f" containerName="kserve-container"
Sep 18 20:06:06.965656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.965404    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerName="kube-rbac-proxy"
Sep 18 20:06:06.969317 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.969294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:06.971379 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.971360    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:06:06.971546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.971528    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:06:06.976938 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.976910    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:06:06.999001 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:06.998977    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:06:07.002319 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.002300    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.004287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.004265    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:06:07.011981 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.011963    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:06:07.134460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.134641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134465    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.134641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.134641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvtpm\" (UniqueName: \"kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.134641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.134641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.134947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.134947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22652\" (UniqueName: \"kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.134947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.134947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.134783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtpm\" (UniqueName: \"kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.236158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22652\" (UniqueName: \"kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.236504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236814 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.236874 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.237021 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.236991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.237094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.237015    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.237094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.237048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.237244 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.237227    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.238950 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.238931    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.239043 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.238957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.243110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.243081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtpm\" (UniqueName: \"kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.243512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.243496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22652\" (UniqueName: \"kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652\") pod \"isvc-canary-promote-predictor-cb75746f8-74sqd\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.282503 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.282478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:07.311969 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.311947    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:07.397171 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.397140    2573 generic.go:356] "Generic (PLEG): container finished" podID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" containerID="b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686" exitCode=0
Sep 18 20:06:07.397392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.397212    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"
Sep 18 20:06:07.397392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.397254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerDied","Data":"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"}
Sep 18 20:06:07.397392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.397313    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn" event={"ID":"41ed148d-ef6a-4274-a05d-9c2c508ab3f8","Type":"ContainerDied","Data":"1f664ef3b5cc6c539555a44dca5ce9a762287abaa3d3cba8317a156a6619efdf"}
Sep 18 20:06:07.397392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.397352    2573 scope.go:117] "RemoveContainer" containerID="979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"
Sep 18 20:06:07.399485 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.399463    2573 generic.go:356] "Generic (PLEG): container finished" podID="64f06168-f642-42fc-9eef-687919d50d31" containerID="27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309" exitCode=0
Sep 18 20:06:07.399595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.399509    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"
Sep 18 20:06:07.399595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.399546    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerDied","Data":"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"}
Sep 18 20:06:07.399677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.399613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s" event={"ID":"64f06168-f642-42fc-9eef-687919d50d31","Type":"ContainerDied","Data":"d5d1c5c509cc1e66cfa682b40f6ed8d02f7a962c7fc7a0ad52abea1f3f023769"}
Sep 18 20:06:07.407352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.407336    2573 scope.go:117] "RemoveContainer" containerID="b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"
Sep 18 20:06:07.415141 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.415000    2573 scope.go:117] "RemoveContainer" containerID="811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"
Sep 18 20:06:07.421812 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.421791    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:06:07.422444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.422432    2573 scope.go:117] "RemoveContainer" containerID="979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"
Sep 18 20:06:07.422694 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.422676    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3\": container with ID starting with 979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3 not found: ID does not exist" containerID="979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"
Sep 18 20:06:07.422905 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.422701    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3"} err="failed to get container status \"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3\": rpc error: code = NotFound desc = could not find container \"979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3\": container with ID starting with 979cf5b802a0cd370afea03ab06ef7169f5aab950081cad31227c337154fbcf3 not found: ID does not exist"
Sep 18 20:06:07.422905 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.422720    2573 scope.go:117] "RemoveContainer" containerID="b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"
Sep 18 20:06:07.423478 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.423257    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686\": container with ID starting with b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686 not found: ID does not exist" containerID="b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"
Sep 18 20:06:07.423478 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.423459    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686"} err="failed to get container status \"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686\": rpc error: code = NotFound desc = could not find container \"b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686\": container with ID starting with b3de68933c152b6683bce24eb7327f0100c2afaa9464d8d6a415b3c4f94ab686 not found: ID does not exist"
Sep 18 20:06:07.423654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.423491    2573 scope.go:117] "RemoveContainer" containerID="811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"
Sep 18 20:06:07.423813 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.423794    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184\": container with ID starting with 811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184 not found: ID does not exist" containerID="811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"
Sep 18 20:06:07.423869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.423818    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184"} err="failed to get container status \"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184\": rpc error: code = NotFound desc = could not find container \"811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184\": container with ID starting with 811ad4d44d8da567021bd543549e8e27caefc08a0256bb542ebeac5fb017a184 not found: ID does not exist"
Sep 18 20:06:07.423869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.423835    2573 scope.go:117] "RemoveContainer" containerID="89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"
Sep 18 20:06:07.425318 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.425300    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-fjw2s"]
Sep 18 20:06:07.431051 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.431035    2573 scope.go:117] "RemoveContainer" containerID="27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"
Sep 18 20:06:07.435343 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.435324    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:06:07.438455 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.438433    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-sf9gn"]
Sep 18 20:06:07.438930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.438914    2573 scope.go:117] "RemoveContainer" containerID="e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"
Sep 18 20:06:07.445769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.445755    2573 scope.go:117] "RemoveContainer" containerID="89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"
Sep 18 20:06:07.446031 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.446014    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232\": container with ID starting with 89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232 not found: ID does not exist" containerID="89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"
Sep 18 20:06:07.446077 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.446037    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232"} err="failed to get container status \"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232\": rpc error: code = NotFound desc = could not find container \"89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232\": container with ID starting with 89226646324eec12c04d0a2d094af932840f45a087468ec584a72a2da59b5232 not found: ID does not exist"
Sep 18 20:06:07.446077 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.446052    2573 scope.go:117] "RemoveContainer" containerID="27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"
Sep 18 20:06:07.446404 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.446383    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309\": container with ID starting with 27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309 not found: ID does not exist" containerID="27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"
Sep 18 20:06:07.446458 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.446412    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309"} err="failed to get container status \"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309\": rpc error: code = NotFound desc = could not find container \"27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309\": container with ID starting with 27be8e46355faebb2f703db9bbe339b6fbd35fbaa7328d9a8c545b6833629309 not found: ID does not exist"
Sep 18 20:06:07.446458 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.446426    2573 scope.go:117] "RemoveContainer" containerID="e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"
Sep 18 20:06:07.446672 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:06:07.446654    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95\": container with ID starting with e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95 not found: ID does not exist" containerID="e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"
Sep 18 20:06:07.446715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.446676    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95"} err="failed to get container status \"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95\": rpc error: code = NotFound desc = could not find container \"e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95\": container with ID starting with e24fb403051d21da465b83cb320f8b21e0559f5186b71c346e921910c17c3a95 not found: ID does not exist"
Sep 18 20:06:07.553610 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.553587    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:06:07.555212 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:06:07.555177    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed21327_cf7a_402f_9c75_c2798cc2d275.slice/crio-82b6883e7f90e7a6be618dc822a89a58abfbce9268bc863a05160e0e27e628ec WatchSource:0}: Error finding container 82b6883e7f90e7a6be618dc822a89a58abfbce9268bc863a05160e0e27e628ec: Status 404 returned error can't find the container with id 82b6883e7f90e7a6be618dc822a89a58abfbce9268bc863a05160e0e27e628ec
Sep 18 20:06:07.590169 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.590149    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:06:07.592354 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:06:07.592333    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012edb9c_81dd_4a64_b093_1648cc702831.slice/crio-f9ea6873b77a5b139ee6640d1a997db44b029323f11f0bdbb63a52ff26073b51 WatchSource:0}: Error finding container f9ea6873b77a5b139ee6640d1a997db44b029323f11f0bdbb63a52ff26073b51: Status 404 returned error can't find the container with id f9ea6873b77a5b139ee6640d1a997db44b029323f11f0bdbb63a52ff26073b51
Sep 18 20:06:07.784092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.784045    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ed148d-ef6a-4274-a05d-9c2c508ab3f8" path="/var/lib/kubelet/pods/41ed148d-ef6a-4274-a05d-9c2c508ab3f8/volumes"
Sep 18 20:06:07.784653 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:07.784631    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f06168-f642-42fc-9eef-687919d50d31" path="/var/lib/kubelet/pods/64f06168-f642-42fc-9eef-687919d50d31/volumes"
Sep 18 20:06:08.405938 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:08.405899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerStarted","Data":"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"}
Sep 18 20:06:08.406132 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:08.405970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerStarted","Data":"f9ea6873b77a5b139ee6640d1a997db44b029323f11f0bdbb63a52ff26073b51"}
Sep 18 20:06:08.407481 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:08.407444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerStarted","Data":"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"}
Sep 18 20:06:08.407599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:08.407492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerStarted","Data":"82b6883e7f90e7a6be618dc822a89a58abfbce9268bc863a05160e0e27e628ec"}
Sep 18 20:06:11.423306 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:11.423280    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerID="165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04" exitCode=0
Sep 18 20:06:11.423668 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:11.423368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerDied","Data":"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"}
Sep 18 20:06:11.425215 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:11.425176    2573 generic.go:356] "Generic (PLEG): container finished" podID="012edb9c-81dd-4a64-b093-1648cc702831" containerID="353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7" exitCode=0
Sep 18 20:06:11.425305 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:11.425225    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerDied","Data":"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"}
Sep 18 20:06:12.431972 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.431936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerStarted","Data":"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"}
Sep 18 20:06:12.432416 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.432013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerStarted","Data":"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"}
Sep 18 20:06:12.432416 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.432392    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:12.432595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.432579    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:12.433964 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.433931    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:12.434523 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.434502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerStarted","Data":"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"}
Sep 18 20:06:12.434635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.434555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerStarted","Data":"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"}
Sep 18 20:06:12.434763 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.434749    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:12.434843 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.434797    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:12.435861 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.435838    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:12.465411 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.465357    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podStartSLOduration=2.765628444 podStartE2EDuration="6.46534636s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699717916s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:12.450114897 +0000 UTC m=+1680.968084528" watchObservedRunningTime="2026-09-18 20:06:12.46534636 +0000 UTC m=+1680.983316000"
Sep 18 20:06:12.482096 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:12.482057    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podStartSLOduration=2.906742719 podStartE2EDuration="6.482044067s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575301348s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:12.479900606 +0000 UTC m=+1680.997870245" watchObservedRunningTime="2026-09-18 20:06:12.482044067 +0000 UTC m=+1681.000013707"
Sep 18 20:06:13.438879 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:13.438828    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:13.439336 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:13.438966    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:18.443128 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:18.443098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:06:18.443682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:18.443497    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:06:18.443682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:18.443570    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:18.443927 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:18.443902    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:28.443632 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:28.443547    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:28.444018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:28.443815    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:38.443727 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:38.443687    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:38.444116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:38.443824    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:48.444049 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:48.444003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:06:48.444049 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:48.444023    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:56.120010 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.119973    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:06:56.124531 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.124512    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.126620 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.126595    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 18 20:06:56.126620 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.126595    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 18 20:06:56.132764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.132737    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:06:56.234271 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.234242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.234412 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.234283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxjt\" (UniqueName: \"kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.234412 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.234314    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.234412 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.234361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.234412 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.234399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335433 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335402    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxjt\" (UniqueName: \"kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.335909 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.335878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.336191 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.336171    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.336305 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.336277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.337864 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.337841    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.343522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.343502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxjt\" (UniqueName: \"kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.436184 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.436114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:06:56.706889 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:56.706865    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:06:56.708896 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:06:56.708870    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6413ae_b113_4edd_b61a_e508f219da0d.slice/crio-5bb814ca0354856bd11b594771e94c8116fdf46b0bdb3345ff1f74e708416f56 WatchSource:0}: Error finding container 5bb814ca0354856bd11b594771e94c8116fdf46b0bdb3345ff1f74e708416f56: Status 404 returned error can't find the container with id 5bb814ca0354856bd11b594771e94c8116fdf46b0bdb3345ff1f74e708416f56
Sep 18 20:06:57.621730 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:57.621696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerStarted","Data":"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"}
Sep 18 20:06:57.622106 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:57.621770    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerStarted","Data":"5bb814ca0354856bd11b594771e94c8116fdf46b0bdb3345ff1f74e708416f56"}
Sep 18 20:06:58.444073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:58.444037    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:06:58.444281 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:06:58.444037    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:07:00.637357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:00.637327    2573 generic.go:356] "Generic (PLEG): container finished" podID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerID="0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67" exitCode=0
Sep 18 20:07:00.637712 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:00.637364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerDied","Data":"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"}
Sep 18 20:07:01.644330 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.644295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerStarted","Data":"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"}
Sep 18 20:07:01.644723 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.644382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerStarted","Data":"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"}
Sep 18 20:07:01.644723 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.644681    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:07:01.644859 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.644819    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:07:01.646149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.646121    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:01.663357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:01.663313    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podStartSLOduration=2.108527698 podStartE2EDuration="5.66329975s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554772052s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:01.660917729 +0000 UTC m=+1730.178887370" watchObservedRunningTime="2026-09-18 20:07:01.66329975 +0000 UTC m=+1730.181269389"
Sep 18 20:07:02.648988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:02.648949    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:03.654129 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:03.654094    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:08.443599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:08.443554    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:07:08.444012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:08.443927    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:07:08.658231 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:08.658188    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:07:08.658796 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:08.658772    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:18.444436 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:18.444391    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:07:18.444815 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:18.444391    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:07:18.659179 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:18.659138    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:28.444141 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:28.444111    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:07:28.444770 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:28.444751    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:07:28.658753 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:28.658710    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:36.998626 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:36.998596    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:07:36.999034 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:36.998891    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" containerID="cri-o://6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb" gracePeriod=30
Sep 18 20:07:36.999034 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:36.998899    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kube-rbac-proxy" containerID="cri-o://fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f" gracePeriod=30
Sep 18 20:07:37.797016 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:37.796982    2573 generic.go:356] "Generic (PLEG): container finished" podID="012edb9c-81dd-4a64-b093-1648cc702831" containerID="fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f" exitCode=2
Sep 18 20:07:37.797183 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:37.797036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerDied","Data":"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"}
Sep 18 20:07:38.439373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:38.439329    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 20:07:38.443621 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:38.443590    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 20:07:38.659053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:38.659016    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:41.539738 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.539720    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:07:41.595543 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595521    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls\") pod \"012edb9c-81dd-4a64-b093-1648cc702831\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") "
Sep 18 20:07:41.595682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595559    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert\") pod \"012edb9c-81dd-4a64-b093-1648cc702831\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") "
Sep 18 20:07:41.595682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location\") pod \"012edb9c-81dd-4a64-b093-1648cc702831\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") "
Sep 18 20:07:41.595682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595608    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22652\" (UniqueName: \"kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652\") pod \"012edb9c-81dd-4a64-b093-1648cc702831\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") "
Sep 18 20:07:41.595682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595640    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config\") pod \"012edb9c-81dd-4a64-b093-1648cc702831\" (UID: \"012edb9c-81dd-4a64-b093-1648cc702831\") "
Sep 18 20:07:41.595903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595882    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "012edb9c-81dd-4a64-b093-1648cc702831" (UID: "012edb9c-81dd-4a64-b093-1648cc702831"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:41.596048 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.595945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "012edb9c-81dd-4a64-b093-1648cc702831" (UID: "012edb9c-81dd-4a64-b093-1648cc702831"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:41.596117 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.596075    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "012edb9c-81dd-4a64-b093-1648cc702831" (UID: "012edb9c-81dd-4a64-b093-1648cc702831"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:41.597663 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.597635    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "012edb9c-81dd-4a64-b093-1648cc702831" (UID: "012edb9c-81dd-4a64-b093-1648cc702831"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:41.597763 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.597677    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652" (OuterVolumeSpecName: "kube-api-access-22652") pod "012edb9c-81dd-4a64-b093-1648cc702831" (UID: "012edb9c-81dd-4a64-b093-1648cc702831"). InnerVolumeSpecName "kube-api-access-22652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:41.696614 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.696587    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012edb9c-81dd-4a64-b093-1648cc702831-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:07:41.696614 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.696611    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:07:41.696772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.696627    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012edb9c-81dd-4a64-b093-1648cc702831-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:07:41.696772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.696639    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22652\" (UniqueName: \"kubernetes.io/projected/012edb9c-81dd-4a64-b093-1648cc702831-kube-api-access-22652\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:07:41.696772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.696651    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012edb9c-81dd-4a64-b093-1648cc702831-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:07:41.817592 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.817558    2573 generic.go:356] "Generic (PLEG): container finished" podID="012edb9c-81dd-4a64-b093-1648cc702831" containerID="6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb" exitCode=0
Sep 18 20:07:41.817714 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.817620    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"
Sep 18 20:07:41.817714 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.817609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerDied","Data":"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"}
Sep 18 20:07:41.817853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.817765    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd" event={"ID":"012edb9c-81dd-4a64-b093-1648cc702831","Type":"ContainerDied","Data":"f9ea6873b77a5b139ee6640d1a997db44b029323f11f0bdbb63a52ff26073b51"}
Sep 18 20:07:41.817853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.817805    2573 scope.go:117] "RemoveContainer" containerID="fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"
Sep 18 20:07:41.826236 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.826220    2573 scope.go:117] "RemoveContainer" containerID="6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"
Sep 18 20:07:41.834231 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.834183    2573 scope.go:117] "RemoveContainer" containerID="353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"
Sep 18 20:07:41.834921 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.834900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:07:41.840189 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.840167    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-74sqd"]
Sep 18 20:07:41.842140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842126    2573 scope.go:117] "RemoveContainer" containerID="fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"
Sep 18 20:07:41.842445 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:07:41.842418    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f\": container with ID starting with fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f not found: ID does not exist" containerID="fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"
Sep 18 20:07:41.842530 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842458    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f"} err="failed to get container status \"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f\": rpc error: code = NotFound desc = could not find container \"fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f\": container with ID starting with fe5c72039922ca455a45066f015bbc88c5653829cad129cf92c6145ec835007f not found: ID does not exist"
Sep 18 20:07:41.842530 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842482    2573 scope.go:117] "RemoveContainer" containerID="6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"
Sep 18 20:07:41.842732 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:07:41.842715    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb\": container with ID starting with 6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb not found: ID does not exist" containerID="6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"
Sep 18 20:07:41.842779 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842739    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb"} err="failed to get container status \"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb\": rpc error: code = NotFound desc = could not find container \"6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb\": container with ID starting with 6069968c0528d80912e5f187c09705bdb7b37d520f097f68657c649cd96e73fb not found: ID does not exist"
Sep 18 20:07:41.842779 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842755    2573 scope.go:117] "RemoveContainer" containerID="353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"
Sep 18 20:07:41.842975 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:07:41.842959    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7\": container with ID starting with 353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7 not found: ID does not exist" containerID="353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"
Sep 18 20:07:41.843019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:41.842980    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7"} err="failed to get container status \"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7\": rpc error: code = NotFound desc = could not find container \"353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7\": container with ID starting with 353df5ac2491906f6965282d2addce5b82776fd70145e0dbe36c0b9f62e4d2f7 not found: ID does not exist"
Sep 18 20:07:43.782730 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:43.782700    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012edb9c-81dd-4a64-b093-1648cc702831" path="/var/lib/kubelet/pods/012edb9c-81dd-4a64-b093-1648cc702831/volumes"
Sep 18 20:07:48.659432 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:48.659395    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:07:58.658778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:07:58.658696    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:08:08.658747 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:08.658707    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:08:18.659349 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:18.659319    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:08:26.241106 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:26.241076    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:08:26.241572 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:26.241435    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" containerID="cri-o://51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b" gracePeriod=30
Sep 18 20:08:26.241572 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:26.241500    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kube-rbac-proxy" containerID="cri-o://b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965" gracePeriod=30
Sep 18 20:08:27.002954 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:27.002919    2573 generic.go:356] "Generic (PLEG): container finished" podID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerID="b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965" exitCode=2
Sep 18 20:08:27.003133 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:27.002998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerDied","Data":"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"}
Sep 18 20:08:28.655110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:28.655071    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 18 20:08:28.659417 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:28.659396    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 20:08:30.476831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.476807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:08:30.585683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.585615    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxjt\" (UniqueName: \"kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt\") pod \"7c6413ae-b113-4edd-b61a-e508f219da0d\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") "
Sep 18 20:08:30.585683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.585673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config\") pod \"7c6413ae-b113-4edd-b61a-e508f219da0d\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") "
Sep 18 20:08:30.585897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.585700    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls\") pod \"7c6413ae-b113-4edd-b61a-e508f219da0d\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") "
Sep 18 20:08:30.585897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.585815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location\") pod \"7c6413ae-b113-4edd-b61a-e508f219da0d\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") "
Sep 18 20:08:30.585897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.585855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert\") pod \"7c6413ae-b113-4edd-b61a-e508f219da0d\" (UID: \"7c6413ae-b113-4edd-b61a-e508f219da0d\") "
Sep 18 20:08:30.586146 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.586125    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c6413ae-b113-4edd-b61a-e508f219da0d" (UID: "7c6413ae-b113-4edd-b61a-e508f219da0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:30.586233 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.586175    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c6413ae-b113-4edd-b61a-e508f219da0d" (UID: "7c6413ae-b113-4edd-b61a-e508f219da0d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:08:30.586310 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.586283    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c6413ae-b113-4edd-b61a-e508f219da0d" (UID: "7c6413ae-b113-4edd-b61a-e508f219da0d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:08:30.587766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.587740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt" (OuterVolumeSpecName: "kube-api-access-6hxjt") pod "7c6413ae-b113-4edd-b61a-e508f219da0d" (UID: "7c6413ae-b113-4edd-b61a-e508f219da0d"). InnerVolumeSpecName "kube-api-access-6hxjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:30.587766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.587746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c6413ae-b113-4edd-b61a-e508f219da0d" (UID: "7c6413ae-b113-4edd-b61a-e508f219da0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:30.686507 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.686484    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxjt\" (UniqueName: \"kubernetes.io/projected/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-api-access-6hxjt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:08:30.686507 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.686504    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:08:30.686652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.686515    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c6413ae-b113-4edd-b61a-e508f219da0d-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:08:30.686652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.686524    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c6413ae-b113-4edd-b61a-e508f219da0d-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:08:30.686652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:30.686533    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c6413ae-b113-4edd-b61a-e508f219da0d-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:08:31.021626 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.021595    2573 generic.go:356] "Generic (PLEG): container finished" podID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerID="51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b" exitCode=0
Sep 18 20:08:31.021799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.021662    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"
Sep 18 20:08:31.021799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.021673    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerDied","Data":"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"}
Sep 18 20:08:31.021799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.021745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8" event={"ID":"7c6413ae-b113-4edd-b61a-e508f219da0d","Type":"ContainerDied","Data":"5bb814ca0354856bd11b594771e94c8116fdf46b0bdb3345ff1f74e708416f56"}
Sep 18 20:08:31.021799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.021782    2573 scope.go:117] "RemoveContainer" containerID="b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"
Sep 18 20:08:31.029863 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.029848    2573 scope.go:117] "RemoveContainer" containerID="51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"
Sep 18 20:08:31.036859 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.036842    2573 scope.go:117] "RemoveContainer" containerID="0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"
Sep 18 20:08:31.042890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.042870    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:08:31.044741 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.044717    2573 scope.go:117] "RemoveContainer" containerID="b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"
Sep 18 20:08:31.045042 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:08:31.045022    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965\": container with ID starting with b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965 not found: ID does not exist" containerID="b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"
Sep 18 20:08:31.045117 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.045051    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965"} err="failed to get container status \"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965\": rpc error: code = NotFound desc = could not find container \"b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965\": container with ID starting with b7d172834eab547ec4001911c7b42e2951116bb326a437ccffc273e296bf8965 not found: ID does not exist"
Sep 18 20:08:31.045117 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.045067    2573 scope.go:117] "RemoveContainer" containerID="51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"
Sep 18 20:08:31.045396 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:08:31.045373    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b\": container with ID starting with 51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b not found: ID does not exist" containerID="51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"
Sep 18 20:08:31.045454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.045406    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b"} err="failed to get container status \"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b\": rpc error: code = NotFound desc = could not find container \"51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b\": container with ID starting with 51ed52885eb7273217f4b4b51e3e621c8c5369d792f00844754d7932e16ffd6b not found: ID does not exist"
Sep 18 20:08:31.045454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.045430    2573 scope.go:117] "RemoveContainer" containerID="0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"
Sep 18 20:08:31.045708 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:08:31.045686    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67\": container with ID starting with 0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67 not found: ID does not exist" containerID="0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"
Sep 18 20:08:31.045792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.045713    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67"} err="failed to get container status \"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67\": rpc error: code = NotFound desc = could not find container \"0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67\": container with ID starting with 0e42df07634635ed347334b2b2a10c3c7543ba2f7439778db41266241eb56a67 not found: ID does not exist"
Sep 18 20:08:31.046282 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.046263    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-7djg8"]
Sep 18 20:08:31.782136 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:08:31.782105    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" path="/var/lib/kubelet/pods/7c6413ae-b113-4edd-b61a-e508f219da0d/volumes"
Sep 18 20:09:07.235940 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:07.235903    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:09:07.236446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:07.236137    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" containerID="cri-o://3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371" gracePeriod=30
Sep 18 20:09:07.236446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:07.236187    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kube-rbac-proxy" containerID="cri-o://163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e" gracePeriod=30
Sep 18 20:09:08.175289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:08.175255    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerID="163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e" exitCode=2
Sep 18 20:09:08.175467 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:08.175327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerDied","Data":"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"}
Sep 18 20:09:08.439915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:08.439818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 20:09:08.444207 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:08.444173    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 20:09:12.176906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.176882    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:09:12.200222 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.200173    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerID="3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371" exitCode=0
Sep 18 20:09:12.200366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.200240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerDied","Data":"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"}
Sep 18 20:09:12.200366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.200258    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"
Sep 18 20:09:12.200366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.200342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl" event={"ID":"6ed21327-cf7a-402f-9c75-c2798cc2d275","Type":"ContainerDied","Data":"82b6883e7f90e7a6be618dc822a89a58abfbce9268bc863a05160e0e27e628ec"}
Sep 18 20:09:12.200486 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.200381    2573 scope.go:117] "RemoveContainer" containerID="163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"
Sep 18 20:09:12.208564 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.208544    2573 scope.go:117] "RemoveContainer" containerID="3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"
Sep 18 20:09:12.216127 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.216101    2573 scope.go:117] "RemoveContainer" containerID="165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"
Sep 18 20:09:12.223352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.223333    2573 scope.go:117] "RemoveContainer" containerID="163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"
Sep 18 20:09:12.223615 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:09:12.223596    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e\": container with ID starting with 163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e not found: ID does not exist" containerID="163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"
Sep 18 20:09:12.223678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.223640    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e"} err="failed to get container status \"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e\": rpc error: code = NotFound desc = could not find container \"163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e\": container with ID starting with 163fe5898f501f9672205fa1c5a4f36b90937d3f55731b0a4c17dc6e49bd960e not found: ID does not exist"
Sep 18 20:09:12.223678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.223657    2573 scope.go:117] "RemoveContainer" containerID="3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"
Sep 18 20:09:12.223881 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:09:12.223865    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371\": container with ID starting with 3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371 not found: ID does not exist" containerID="3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"
Sep 18 20:09:12.223925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.223885    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371"} err="failed to get container status \"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371\": rpc error: code = NotFound desc = could not find container \"3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371\": container with ID starting with 3c1e87e665f8b6d879a7cf31389f1060d7445e076e44f0fa4e681d83a95f0371 not found: ID does not exist"
Sep 18 20:09:12.223925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.223897    2573 scope.go:117] "RemoveContainer" containerID="165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"
Sep 18 20:09:12.224092 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:09:12.224075    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04\": container with ID starting with 165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04 not found: ID does not exist" containerID="165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"
Sep 18 20:09:12.224134 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.224098    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04"} err="failed to get container status \"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04\": rpc error: code = NotFound desc = could not find container \"165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04\": container with ID starting with 165a87d4a367ae574188f56b3837df70c4d40d35365096f0517ddf5e364d1e04 not found: ID does not exist"
Sep 18 20:09:12.235528 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235510    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvtpm\" (UniqueName: \"kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm\") pod \"6ed21327-cf7a-402f-9c75-c2798cc2d275\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") "
Sep 18 20:09:12.235627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235537    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location\") pod \"6ed21327-cf7a-402f-9c75-c2798cc2d275\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") "
Sep 18 20:09:12.235627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235572    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config\") pod \"6ed21327-cf7a-402f-9c75-c2798cc2d275\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") "
Sep 18 20:09:12.235627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235590    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert\") pod \"6ed21327-cf7a-402f-9c75-c2798cc2d275\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") "
Sep 18 20:09:12.235792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235642    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls\") pod \"6ed21327-cf7a-402f-9c75-c2798cc2d275\" (UID: \"6ed21327-cf7a-402f-9c75-c2798cc2d275\") "
Sep 18 20:09:12.235889 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235863    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ed21327-cf7a-402f-9c75-c2798cc2d275" (UID: "6ed21327-cf7a-402f-9c75-c2798cc2d275"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:09:12.235945 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235874    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ed21327-cf7a-402f-9c75-c2798cc2d275" (UID: "6ed21327-cf7a-402f-9c75-c2798cc2d275"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:12.235987 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.235946    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ed21327-cf7a-402f-9c75-c2798cc2d275" (UID: "6ed21327-cf7a-402f-9c75-c2798cc2d275"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:09:12.237586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.237562    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm" (OuterVolumeSpecName: "kube-api-access-dvtpm") pod "6ed21327-cf7a-402f-9c75-c2798cc2d275" (UID: "6ed21327-cf7a-402f-9c75-c2798cc2d275"). InnerVolumeSpecName "kube-api-access-dvtpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:09:12.237748 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.237730    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ed21327-cf7a-402f-9c75-c2798cc2d275" (UID: "6ed21327-cf7a-402f-9c75-c2798cc2d275"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:09:12.336536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.336438    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvtpm\" (UniqueName: \"kubernetes.io/projected/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-api-access-dvtpm\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.336536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.336469    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ed21327-cf7a-402f-9c75-c2798cc2d275-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.336536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.336484    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.336536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.336496    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ed21327-cf7a-402f-9c75-c2798cc2d275-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.336536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.336512    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ed21327-cf7a-402f-9c75-c2798cc2d275-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:09:12.442102 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442073    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:09:12.442479 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442461    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kube-rbac-proxy"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442482    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kube-rbac-proxy"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442495    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kube-rbac-proxy"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442504    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kube-rbac-proxy"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442515    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442524    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442538    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="storage-initializer"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442546    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="storage-initializer"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442555    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442564    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container"
Sep 18 20:09:12.442578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442576    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kube-rbac-proxy"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442585    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kube-rbac-proxy"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442595    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="storage-initializer"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442603    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="storage-initializer"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442627    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442636    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442652    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="storage-initializer"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442663    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="storage-initializer"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442732    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kserve-container"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442745    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kserve-container"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442755    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kserve-container"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442766    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="012edb9c-81dd-4a64-b093-1648cc702831" containerName="kube-rbac-proxy"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442777    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6413ae-b113-4edd-b61a-e508f219da0d" containerName="kube-rbac-proxy"
Sep 18 20:09:12.443076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.442786    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" containerName="kube-rbac-proxy"
Sep 18 20:09:12.447717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.447696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.449773 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.449736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 20:09:12.449941 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.449917    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 20:09:12.454658 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.454489    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:09:12.478027 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.477991    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:09:12.481754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.481728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.484172 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.484152    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 20:09:12.490582 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.490557    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:09:12.526457 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.526421    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:09:12.537655 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk2k\" (UniqueName: \"kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.537837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.537837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.537837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-684hf\" (UniqueName: \"kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.538007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.537999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.538352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.538316    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-l2xpl"]
Sep 18 20:09:12.638750 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk2k\" (UniqueName: \"kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638799    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-684hf\" (UniqueName: \"kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.638959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.639397 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.638978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.639397 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.639397 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639364    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.639531 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:09:12.639375    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 20:09:12.639531 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.639621 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:09:12.639516    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls podName:2eff3092-c865-4791-b4bf-70316f26dc83 nodeName:}" failed. No retries permitted until 2026-09-18 20:09:13.139495416 +0000 UTC m=+1861.657465039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 20:09:12.639690 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639668    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.639727 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.639770 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639746    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.639853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.639832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.641450 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.641431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.647548 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.647524    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk2k\" (UniqueName: \"kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-6k589\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:12.647656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.647530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-684hf\" (UniqueName: \"kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:12.792802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:12.792764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:13.067099 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.067067    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:09:13.069149 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:09:13.069127    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6eedbd_6da3_4d05_b3a0_b82deb741c2d.slice/crio-9aac2639b455d65c5c32ea2c0d7da5647ef61a0757c053463a86bb4fa827fa4b WatchSource:0}: Error finding container 9aac2639b455d65c5c32ea2c0d7da5647ef61a0757c053463a86bb4fa827fa4b: Status 404 returned error can't find the container with id 9aac2639b455d65c5c32ea2c0d7da5647ef61a0757c053463a86bb4fa827fa4b
Sep 18 20:09:13.143104 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.143059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:13.145446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.145418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-m9f2h\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:13.207069 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.207027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerStarted","Data":"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"}
Sep 18 20:09:13.207534 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.207108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerStarted","Data":"9aac2639b455d65c5c32ea2c0d7da5647ef61a0757c053463a86bb4fa827fa4b"}
Sep 18 20:09:13.358398 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.358350    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:13.631931 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.631901    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:09:13.634559 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:09:13.634530    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eff3092_c865_4791_b4bf_70316f26dc83.slice/crio-52290b00c00c13a4655b9867094e1b03241209256ad57e1b1298c9aeda669d89 WatchSource:0}: Error finding container 52290b00c00c13a4655b9867094e1b03241209256ad57e1b1298c9aeda669d89: Status 404 returned error can't find the container with id 52290b00c00c13a4655b9867094e1b03241209256ad57e1b1298c9aeda669d89
Sep 18 20:09:13.782421 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:13.782373    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed21327-cf7a-402f-9c75-c2798cc2d275" path="/var/lib/kubelet/pods/6ed21327-cf7a-402f-9c75-c2798cc2d275/volumes"
Sep 18 20:09:14.216408 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:14.216359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerStarted","Data":"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"}
Sep 18 20:09:14.216840 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:14.216441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerStarted","Data":"52290b00c00c13a4655b9867094e1b03241209256ad57e1b1298c9aeda669d89"}
Sep 18 20:09:17.229528 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:17.229488    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerID="b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753" exitCode=0
Sep 18 20:09:17.229949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:17.229562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerDied","Data":"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"}
Sep 18 20:09:18.236706 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.236659    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerStarted","Data":"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"}
Sep 18 20:09:18.237186 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.236736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerStarted","Data":"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"}
Sep 18 20:09:18.237186 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.236998    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:18.238188 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.238165    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eff3092-c865-4791-b4bf-70316f26dc83" containerID="e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43" exitCode=0
Sep 18 20:09:18.238311 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.238226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerDied","Data":"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"}
Sep 18 20:09:18.275783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.275738    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podStartSLOduration=2.347056953 podStartE2EDuration="6.275719295s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.928662342s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:18.255257224 +0000 UTC m=+1866.773226864" watchObservedRunningTime="2026-09-18 20:09:18.275719295 +0000 UTC m=+1866.793688948"
Sep 18 20:09:18.373900 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.373872    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:09:18.377347 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.377326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.379461 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.379436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:09:18.379573 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.379471    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 18 20:09:18.387030 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.387009    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:09:18.492902 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.492804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.492902 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.492870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.492902 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.492896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.493149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.492920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz96n\" (UniqueName: \"kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.493149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.492958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594128 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz96n\" (UniqueName: \"kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594539 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.594961 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.595017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.594947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.596621 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.596601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.602570 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.602551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz96n\" (UniqueName: \"kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-cshjb\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.688071 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.688035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:18.964691 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:18.964664    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:09:18.966773 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:09:18.966744    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eb06fc_108b_4749_a3f4_56073a8c2903.slice/crio-ca1be8d8c5e3aad31f7bfafacfce44ad73f4f69e394cd47bbe7dc647487cc377 WatchSource:0}: Error finding container ca1be8d8c5e3aad31f7bfafacfce44ad73f4f69e394cd47bbe7dc647487cc377: Status 404 returned error can't find the container with id ca1be8d8c5e3aad31f7bfafacfce44ad73f4f69e394cd47bbe7dc647487cc377
Sep 18 20:09:19.245678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.245574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerStarted","Data":"785b33bd0a8f30147ad93f7d49f86922de28b0979232764793a6c27cfcd96e25"}
Sep 18 20:09:19.245678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.245655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerStarted","Data":"ca1be8d8c5e3aad31f7bfafacfce44ad73f4f69e394cd47bbe7dc647487cc377"}
Sep 18 20:09:19.248068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.248040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerStarted","Data":"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"}
Sep 18 20:09:19.248231 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.248100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerStarted","Data":"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"}
Sep 18 20:09:19.248312 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.248293    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:19.248563 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.248547    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:19.248650 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.248589    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:19.249949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.249920    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:19.249949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.249937    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:09:19.280922 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:19.280863    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podStartSLOduration=3.238195194 podStartE2EDuration="7.280846378s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.042651184s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:19.279454762 +0000 UTC m=+1867.797424413" watchObservedRunningTime="2026-09-18 20:09:19.280846378 +0000 UTC m=+1867.798816019"
Sep 18 20:09:20.255398 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:20.255352    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:09:20.256054 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:20.256022    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:24.271361 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:24.271328    2573 generic.go:356] "Generic (PLEG): container finished" podID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerID="785b33bd0a8f30147ad93f7d49f86922de28b0979232764793a6c27cfcd96e25" exitCode=0
Sep 18 20:09:24.271915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:24.271400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerDied","Data":"785b33bd0a8f30147ad93f7d49f86922de28b0979232764793a6c27cfcd96e25"}
Sep 18 20:09:24.272694 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:24.272678    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:09:25.260412 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:25.260181    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:09:25.260910 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:25.260711    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:09:25.260910 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:25.260723    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:25.261433 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:25.261404    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:09:28.294430 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:28.294335    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerStarted","Data":"6d49ae9e344427b8b6ba8a01dcf4386a15e1bf43ebd60a3402ff95ecefb2c4c1"}
Sep 18 20:09:28.294430 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:28.294406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerStarted","Data":"b0271c927741861f855bc547a3503c58abf44b80ce1b79c5ced325c9124d9c2d"}
Sep 18 20:09:28.294811 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:28.294618    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:28.312679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:28.312628    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podStartSLOduration=1.972133208 podStartE2EDuration="10.312612973s" totalImagesPullingTime="3.684275865s" totalInitContainerRuntime="4.6562039s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:28.311443943 +0000 UTC m=+1876.829413585" watchObservedRunningTime="2026-09-18 20:09:28.312612973 +0000 UTC m=+1876.830582613"
Sep 18 20:09:29.298166 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:29.298129    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:29.299461 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:29.299431    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:09:30.302836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:30.302792    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:09:35.260972 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:35.260928    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:35.261460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:35.261243    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:09:35.307818 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:35.307788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:35.308238 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:35.308192    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 20:09:45.261176 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:45.261126    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:45.261637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:45.261231    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:09:45.308618 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:45.308590    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:09:55.261436 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:55.261400    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:09:55.261907 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:09:55.261397    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:00.147499 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.147461    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:10:00.147973 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.147772    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container" containerID="cri-o://b0271c927741861f855bc547a3503c58abf44b80ce1b79c5ced325c9124d9c2d" gracePeriod=30
Sep 18 20:10:00.147973 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.147862    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" containerID="cri-o://6d49ae9e344427b8b6ba8a01dcf4386a15e1bf43ebd60a3402ff95ecefb2c4c1" gracePeriod=30
Sep 18 20:10:00.304073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.304027    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:00.433900 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.433799    2573 generic.go:356] "Generic (PLEG): container finished" podID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerID="6d49ae9e344427b8b6ba8a01dcf4386a15e1bf43ebd60a3402ff95ecefb2c4c1" exitCode=2
Sep 18 20:10:00.433900 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:00.433867    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerDied","Data":"6d49ae9e344427b8b6ba8a01dcf4386a15e1bf43ebd60a3402ff95ecefb2c4c1"}
Sep 18 20:10:05.260950 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:05.260909    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:10:05.261445 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:05.261411    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:05.304071 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:05.304027    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:10.303065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:10.303026    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:10.303477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:10.303148    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:10:15.261490 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:15.261449    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:10:15.261895 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:15.261449    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:15.303897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:15.303859    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:20.303258 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:20.303195    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:25.261508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:25.261469    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:25.261895 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:25.261469    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:10:25.303423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:25.303385    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:25.777687 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:25.777642    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:30.303395 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.303357    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 20:10:30.566334 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.566240    2573 generic.go:356] "Generic (PLEG): container finished" podID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerID="b0271c927741861f855bc547a3503c58abf44b80ce1b79c5ced325c9124d9c2d" exitCode=137
Sep 18 20:10:30.566500 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.566319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerDied","Data":"b0271c927741861f855bc547a3503c58abf44b80ce1b79c5ced325c9124d9c2d"}
Sep 18 20:10:30.783390 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.783368    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:10:30.923162 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config\") pod \"70eb06fc-108b-4749-a3f4-56073a8c2903\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") "
Sep 18 20:10:30.923374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923174    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location\") pod \"70eb06fc-108b-4749-a3f4-56073a8c2903\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") "
Sep 18 20:10:30.923374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923230    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz96n\" (UniqueName: \"kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n\") pod \"70eb06fc-108b-4749-a3f4-56073a8c2903\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") "
Sep 18 20:10:30.923374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923279    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls\") pod \"70eb06fc-108b-4749-a3f4-56073a8c2903\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") "
Sep 18 20:10:30.923374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923307    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert\") pod \"70eb06fc-108b-4749-a3f4-56073a8c2903\" (UID: \"70eb06fc-108b-4749-a3f4-56073a8c2903\") "
Sep 18 20:10:30.923606 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923575    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70eb06fc-108b-4749-a3f4-56073a8c2903" (UID: "70eb06fc-108b-4749-a3f4-56073a8c2903"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:30.923795 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.923775    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70eb06fc-108b-4749-a3f4-56073a8c2903" (UID: "70eb06fc-108b-4749-a3f4-56073a8c2903"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:30.925514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.925478    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70eb06fc-108b-4749-a3f4-56073a8c2903" (UID: "70eb06fc-108b-4749-a3f4-56073a8c2903"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:30.925608 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.925512    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n" (OuterVolumeSpecName: "kube-api-access-nz96n") pod "70eb06fc-108b-4749-a3f4-56073a8c2903" (UID: "70eb06fc-108b-4749-a3f4-56073a8c2903"). InnerVolumeSpecName "kube-api-access-nz96n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:30.938141 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:30.938110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70eb06fc-108b-4749-a3f4-56073a8c2903" (UID: "70eb06fc-108b-4749-a3f4-56073a8c2903"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:31.024028 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.023996    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:31.024028 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.024019    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70eb06fc-108b-4749-a3f4-56073a8c2903-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:31.024028 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.024029    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz96n\" (UniqueName: \"kubernetes.io/projected/70eb06fc-108b-4749-a3f4-56073a8c2903-kube-api-access-nz96n\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:31.024277 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.024039    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70eb06fc-108b-4749-a3f4-56073a8c2903-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:31.024277 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.024049    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70eb06fc-108b-4749-a3f4-56073a8c2903-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:31.572945 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.572911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb" event={"ID":"70eb06fc-108b-4749-a3f4-56073a8c2903","Type":"ContainerDied","Data":"ca1be8d8c5e3aad31f7bfafacfce44ad73f4f69e394cd47bbe7dc647487cc377"}
Sep 18 20:10:31.573409 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.572990    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"
Sep 18 20:10:31.573409 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.573008    2573 scope.go:117] "RemoveContainer" containerID="6d49ae9e344427b8b6ba8a01dcf4386a15e1bf43ebd60a3402ff95ecefb2c4c1"
Sep 18 20:10:31.582461 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.582435    2573 scope.go:117] "RemoveContainer" containerID="b0271c927741861f855bc547a3503c58abf44b80ce1b79c5ced325c9124d9c2d"
Sep 18 20:10:31.590327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.590304    2573 scope.go:117] "RemoveContainer" containerID="785b33bd0a8f30147ad93f7d49f86922de28b0979232764793a6c27cfcd96e25"
Sep 18 20:10:31.595422 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.595401    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:10:31.597323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.597297    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-cshjb"]
Sep 18 20:10:31.782501 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:31.782471    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" path="/var/lib/kubelet/pods/70eb06fc-108b-4749-a3f4-56073a8c2903/volumes"
Sep 18 20:10:35.261369 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:35.261338    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:10:35.777795 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:35.777739    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:45.782653 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:45.782621    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:10:52.437364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:52.437331    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:10:52.437786 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:52.437627    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" containerID="cri-o://cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211" gracePeriod=30
Sep 18 20:10:52.437786 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:52.437679    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kube-rbac-proxy" containerID="cri-o://76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273" gracePeriod=30
Sep 18 20:10:52.676095 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:52.676058    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eff3092-c865-4791-b4bf-70316f26dc83" containerID="76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273" exitCode=2
Sep 18 20:10:52.676287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:52.676128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerDied","Data":"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"}
Sep 18 20:10:55.254137 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:55.254051    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 20:10:55.777704 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:55.777659    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 20:10:57.378703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.378679    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:10:57.450916 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.450880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-684hf\" (UniqueName: \"kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf\") pod \"2eff3092-c865-4791-b4bf-70316f26dc83\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") "
Sep 18 20:10:57.451094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.450947    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert\") pod \"2eff3092-c865-4791-b4bf-70316f26dc83\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") "
Sep 18 20:10:57.451094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.450971    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location\") pod \"2eff3092-c865-4791-b4bf-70316f26dc83\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") "
Sep 18 20:10:57.451094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.451002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") pod \"2eff3092-c865-4791-b4bf-70316f26dc83\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") "
Sep 18 20:10:57.451094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.451025    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config\") pod \"2eff3092-c865-4791-b4bf-70316f26dc83\" (UID: \"2eff3092-c865-4791-b4bf-70316f26dc83\") "
Sep 18 20:10:57.451383 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.451353    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2eff3092-c865-4791-b4bf-70316f26dc83" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:57.451434 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.451375    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2eff3092-c865-4791-b4bf-70316f26dc83" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:57.451487 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.451465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2eff3092-c865-4791-b4bf-70316f26dc83" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:57.453086 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.453064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2eff3092-c865-4791-b4bf-70316f26dc83" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:57.453137 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.453117    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf" (OuterVolumeSpecName: "kube-api-access-684hf") pod "2eff3092-c865-4791-b4bf-70316f26dc83" (UID: "2eff3092-c865-4791-b4bf-70316f26dc83"). InnerVolumeSpecName "kube-api-access-684hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:57.551845 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.551767    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:57.551845 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.551793    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eff3092-c865-4791-b4bf-70316f26dc83-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:57.551845 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.551803    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eff3092-c865-4791-b4bf-70316f26dc83-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:57.551845 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.551813    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eff3092-c865-4791-b4bf-70316f26dc83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:57.551845 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.551821    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-684hf\" (UniqueName: \"kubernetes.io/projected/2eff3092-c865-4791-b4bf-70316f26dc83-kube-api-access-684hf\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:10:57.699944 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.699909    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eff3092-c865-4791-b4bf-70316f26dc83" containerID="cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211" exitCode=0
Sep 18 20:10:57.700140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.699967    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"
Sep 18 20:10:57.700140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.699976    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerDied","Data":"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"}
Sep 18 20:10:57.700140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.700041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h" event={"ID":"2eff3092-c865-4791-b4bf-70316f26dc83","Type":"ContainerDied","Data":"52290b00c00c13a4655b9867094e1b03241209256ad57e1b1298c9aeda669d89"}
Sep 18 20:10:57.700140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.700076    2573 scope.go:117] "RemoveContainer" containerID="76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"
Sep 18 20:10:57.708670 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.708652    2573 scope.go:117] "RemoveContainer" containerID="cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"
Sep 18 20:10:57.716080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.716059    2573 scope.go:117] "RemoveContainer" containerID="e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"
Sep 18 20:10:57.722771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.722749    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:10:57.724240 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.724218    2573 scope.go:117] "RemoveContainer" containerID="76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"
Sep 18 20:10:57.724546 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:10:57.724525    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273\": container with ID starting with 76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273 not found: ID does not exist" containerID="76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"
Sep 18 20:10:57.724617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.724552    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273"} err="failed to get container status \"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273\": rpc error: code = NotFound desc = could not find container \"76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273\": container with ID starting with 76278ec66480ad5e542ab8572c7f284c49b6a44bc0932ffc1431a8f739a3e273 not found: ID does not exist"
Sep 18 20:10:57.724617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.724570    2573 scope.go:117] "RemoveContainer" containerID="cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"
Sep 18 20:10:57.725050 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:10:57.725019    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211\": container with ID starting with cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211 not found: ID does not exist" containerID="cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"
Sep 18 20:10:57.725050 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.725055    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211"} err="failed to get container status \"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211\": rpc error: code = NotFound desc = could not find container \"cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211\": container with ID starting with cb2a87b7ccba4163ca589e8b0da21709754641d807f05c035187ac36da01a211 not found: ID does not exist"
Sep 18 20:10:57.725050 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.725075    2573 scope.go:117] "RemoveContainer" containerID="e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"
Sep 18 20:10:57.725777 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:10:57.725421    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43\": container with ID starting with e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43 not found: ID does not exist" containerID="e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"
Sep 18 20:10:57.725777 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.725455    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43"} err="failed to get container status \"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43\": rpc error: code = NotFound desc = could not find container \"e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43\": container with ID starting with e3d514b94abc3e4ccf485d33d94c615b1b6606b41441fcf7f4ee67d3d476ce43 not found: ID does not exist"
Sep 18 20:10:57.727423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.727402    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-m9f2h"]
Sep 18 20:10:57.782404 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:10:57.782374    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" path="/var/lib/kubelet/pods/2eff3092-c865-4791-b4bf-70316f26dc83/volumes"
Sep 18 20:12:22.790216 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790168    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790511    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790523    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790531    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790536    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790543    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790549    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790563    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790570    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790580    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="storage-initializer"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790587    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="storage-initializer"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790600    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="storage-initializer"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790605    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="storage-initializer"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790659    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kserve-container"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790665    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eff3092-c865-4791-b4bf-70316f26dc83" containerName="kube-rbac-proxy"
Sep 18 20:12:22.790737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.790671    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70eb06fc-108b-4749-a3f4-56073a8c2903" containerName="kserve-container"
Sep 18 20:12:22.793018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.792998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.795332 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.795306    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 20:12:22.795455 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.795357    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 18 20:12:22.804482 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.804460    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:12:22.882713 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.882672    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.882713 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.882717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.883007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.882795    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbsq\" (UniqueName: \"kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.883007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.882863    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.883007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.882941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984077 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbsq\" (UniqueName: \"kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.984640 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.984618    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.985042 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.985021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.985088 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.985021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.986635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.986617    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:22.992104 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:22.992081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbsq\" (UniqueName: \"kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq\") pod \"isvc-xgboost-predictor-7566d65bf7-ptfg8\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:23.103104 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:23.103075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:23.372158 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:23.372131    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:12:23.374279 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:12:23.374249    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ba9025_4999_435c_83bb_ac8c44346357.slice/crio-777661cbb04b38da25c104be4e5a6de15ee63593e2ff9e582fecf58f5e64bcb2 WatchSource:0}: Error finding container 777661cbb04b38da25c104be4e5a6de15ee63593e2ff9e582fecf58f5e64bcb2: Status 404 returned error can't find the container with id 777661cbb04b38da25c104be4e5a6de15ee63593e2ff9e582fecf58f5e64bcb2
Sep 18 20:12:24.051341 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:24.051306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerStarted","Data":"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"}
Sep 18 20:12:24.051724 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:24.051361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerStarted","Data":"777661cbb04b38da25c104be4e5a6de15ee63593e2ff9e582fecf58f5e64bcb2"}
Sep 18 20:12:28.069733 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:28.069697    2573 generic.go:356] "Generic (PLEG): container finished" podID="94ba9025-4999-435c-83bb-ac8c44346357" containerID="0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e" exitCode=0
Sep 18 20:12:28.070113 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:28.069751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerDied","Data":"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"}
Sep 18 20:12:46.158906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:46.158865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerStarted","Data":"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"}
Sep 18 20:12:46.159289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:46.158935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerStarted","Data":"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"}
Sep 18 20:12:46.159289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:46.159122    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:46.177071 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:46.177024    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podStartSLOduration=2.587222563 podStartE2EDuration="24.177011551s" totalImagesPullingTime="17.652555668s" totalInitContainerRuntime="3.93723332s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:46.174943933 +0000 UTC m=+2074.692913574" watchObservedRunningTime="2026-09-18 20:12:46.177011551 +0000 UTC m=+2074.694981188"
Sep 18 20:12:47.163014 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:47.162979    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:47.164101 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:47.164069    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:12:48.167216 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:48.167156    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:12:53.171169 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:53.171140    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:12:53.171796 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:12:53.171758    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:03.172224 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:03.172159    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:12.604687 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:12.604651    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:13:12.605164 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:12.604878    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" containerID="cri-o://c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5" gracePeriod=30
Sep 18 20:13:12.605164 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:12.604930    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kube-rbac-proxy" containerID="cri-o://3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89" gracePeriod=30
Sep 18 20:13:13.172083 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:13.171990    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:13.272867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:13.272833    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerID="3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89" exitCode=2
Sep 18 20:13:13.273029 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:13.272908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerDied","Data":"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"}
Sep 18 20:13:15.255479 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:15.255434    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 20:13:15.260639 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:15.260606    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 20:13:17.921283 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.921247    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:13:17.938485 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.938456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:13:17.938647 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.938587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:17.940981 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.940953    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 20:13:17.941137 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.941098    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 20:13:17.972696 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.972666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdmt\" (UniqueName: \"kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:17.972853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.972697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:17.972853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.972786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:17.972853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.972832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:17.972853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:17.972851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdmt\" (UniqueName: \"kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074313 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074100    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074313 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074313 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074557 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.074815 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.074788    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.075093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.075072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.076747 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.076728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.082283 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.082258    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdmt\" (UniqueName: \"kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt\") pod \"isvc-canary-stop-predictor-5d5b899946-lvdqd\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.186391 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.186317    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:13:18.250976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.250663    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:18.276155 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location\") pod \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") "
Sep 18 20:13:18.276296 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert\") pod \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") "
Sep 18 20:13:18.276296 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276259    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpk2k\" (UniqueName: \"kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k\") pod \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") "
Sep 18 20:13:18.276296 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276277    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls\") pod \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") "
Sep 18 20:13:18.276475 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276320    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config\") pod \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\" (UID: \"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d\") "
Sep 18 20:13:18.276475 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276448    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" (UID: "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:13:18.276589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" (UID: "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:18.276589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276563    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:13:18.276589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276578    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:13:18.276833 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.276803    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" (UID: "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:13:18.278351 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.278328    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" (UID: "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:13:18.278539 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.278520    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k" (OuterVolumeSpecName: "kube-api-access-kpk2k") pod "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" (UID: "2b6eedbd-6da3-4d05-b3a0-b82deb741c2d"). InnerVolumeSpecName "kube-api-access-kpk2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:13:18.295246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.295213    2573 generic.go:356] "Generic (PLEG): container finished" podID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerID="c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5" exitCode=0
Sep 18 20:13:18.295364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.295278    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"
Sep 18 20:13:18.295364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.295297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerDied","Data":"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"}
Sep 18 20:13:18.295444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.295373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589" event={"ID":"2b6eedbd-6da3-4d05-b3a0-b82deb741c2d","Type":"ContainerDied","Data":"9aac2639b455d65c5c32ea2c0d7da5647ef61a0757c053463a86bb4fa827fa4b"}
Sep 18 20:13:18.295444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.295401    2573 scope.go:117] "RemoveContainer" containerID="3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"
Sep 18 20:13:18.304511 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.304490    2573 scope.go:117] "RemoveContainer" containerID="c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"
Sep 18 20:13:18.317977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.317001    2573 scope.go:117] "RemoveContainer" containerID="b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"
Sep 18 20:13:18.320121 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.320028    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:13:18.321914 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.321889    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-6k589"]
Sep 18 20:13:18.331032 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.330917    2573 scope.go:117] "RemoveContainer" containerID="3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"
Sep 18 20:13:18.331633 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:13:18.331536    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89\": container with ID starting with 3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89 not found: ID does not exist" containerID="3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"
Sep 18 20:13:18.331633 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.331577    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89"} err="failed to get container status \"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89\": rpc error: code = NotFound desc = could not find container \"3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89\": container with ID starting with 3cebeb2c9c30554e96430eafe5264c87f96239be0d863a6f1a4b72b07b789a89 not found: ID does not exist"
Sep 18 20:13:18.332047 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.331603    2573 scope.go:117] "RemoveContainer" containerID="c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"
Sep 18 20:13:18.332597 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:13:18.332453    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5\": container with ID starting with c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5 not found: ID does not exist" containerID="c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"
Sep 18 20:13:18.332597 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.332485    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5"} err="failed to get container status \"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5\": rpc error: code = NotFound desc = could not find container \"c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5\": container with ID starting with c6b133d3acaf189426b4fb63bcee1c32a6822a3d4cb159cba313fe9a25c8a3c5 not found: ID does not exist"
Sep 18 20:13:18.332597 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.332506    2573 scope.go:117] "RemoveContainer" containerID="b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"
Sep 18 20:13:18.333175 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:13:18.333154    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753\": container with ID starting with b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753 not found: ID does not exist" containerID="b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"
Sep 18 20:13:18.333266 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.333182    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753"} err="failed to get container status \"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753\": rpc error: code = NotFound desc = could not find container \"b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753\": container with ID starting with b512f4f4e455da73135f766c748f678a8a7922c76cf8cc980abdb1a5431a0753 not found: ID does not exist"
Sep 18 20:13:18.377790 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.377756    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk2k\" (UniqueName: \"kubernetes.io/projected/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-api-access-kpk2k\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:13:18.377790 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.377785    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:13:18.377978 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.377800    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:13:18.535544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:18.535513    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:13:18.538606 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:13:18.538575    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3070b071_f7e8_4abc_877f_02811d4fa750.slice/crio-673ac9d4b7660f968f72e3265b92a9bb6d64c64557eabf6bcf7d655a331b6056 WatchSource:0}: Error finding container 673ac9d4b7660f968f72e3265b92a9bb6d64c64557eabf6bcf7d655a331b6056: Status 404 returned error can't find the container with id 673ac9d4b7660f968f72e3265b92a9bb6d64c64557eabf6bcf7d655a331b6056
Sep 18 20:13:19.301618 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:19.301579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerStarted","Data":"9b571c918bd31544cace1ab49a8065e1643e07224f0875ff61ed168cdff5af4b"}
Sep 18 20:13:19.301990 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:19.301654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerStarted","Data":"673ac9d4b7660f968f72e3265b92a9bb6d64c64557eabf6bcf7d655a331b6056"}
Sep 18 20:13:19.783625 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:19.783591    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" path="/var/lib/kubelet/pods/2b6eedbd-6da3-4d05-b3a0-b82deb741c2d/volumes"
Sep 18 20:13:23.171965 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:23.171925    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:23.319847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:23.319812    2573 generic.go:356] "Generic (PLEG): container finished" podID="3070b071-f7e8-4abc-877f-02811d4fa750" containerID="9b571c918bd31544cace1ab49a8065e1643e07224f0875ff61ed168cdff5af4b" exitCode=0
Sep 18 20:13:23.320019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:23.319883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerDied","Data":"9b571c918bd31544cace1ab49a8065e1643e07224f0875ff61ed168cdff5af4b"}
Sep 18 20:13:24.326328 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:24.326288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerStarted","Data":"084024667bd1b263dd38913a80921afc129d8c9af2d9e0074d8ea216be12dc9e"}
Sep 18 20:13:24.326870 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:24.326411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerStarted","Data":"8ec6ceab8b1e826983eb7d48e7f7cdbf768b5eaa5d72ef2a90bb7b7f76d6f8d4"}
Sep 18 20:13:24.327123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:24.327097    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:24.344776 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:24.344727    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podStartSLOduration=3.626237116 podStartE2EDuration="7.344713755s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.718476639s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:13:24.343558748 +0000 UTC m=+2112.861528385" watchObservedRunningTime="2026-09-18 20:13:24.344713755 +0000 UTC m=+2112.862683394"
Sep 18 20:13:25.330592 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:25.330555    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:25.332291 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:25.332257    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:13:26.335101 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:26.335060    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:13:31.339958 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:31.339927    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:13:31.340595 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:31.340568    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:13:33.171679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:33.171628    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:41.341454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:41.341405    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:13:43.171783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:43.171742    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:13:51.340880 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:51.340840    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:13:53.172095 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:13:53.172060    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 20:14:01.341014 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:01.340934    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:14:03.172346 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:03.172318    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:14:11.340956 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:11.340912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:14:13.425753 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:13.425718    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:14:13.427313 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:13.426067    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kube-rbac-proxy" containerID="cri-o://267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370" gracePeriod=30
Sep 18 20:14:13.427313 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:13.426048    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container" containerID="cri-o://1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397" gracePeriod=30
Sep 18 20:14:14.539896 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:14.539862    2573 generic.go:356] "Generic (PLEG): container finished" podID="94ba9025-4999-435c-83bb-ac8c44346357" containerID="267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370" exitCode=2
Sep 18 20:14:14.540303 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:14.539914    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerDied","Data":"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"}
Sep 18 20:14:17.969525 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:17.969500    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:14:18.071295 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071180    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config\") pod \"94ba9025-4999-435c-83bb-ac8c44346357\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") "
Sep 18 20:14:18.071477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071300    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls\") pod \"94ba9025-4999-435c-83bb-ac8c44346357\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") "
Sep 18 20:14:18.071477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071333    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location\") pod \"94ba9025-4999-435c-83bb-ac8c44346357\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") "
Sep 18 20:14:18.071477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071362    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbsq\" (UniqueName: \"kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq\") pod \"94ba9025-4999-435c-83bb-ac8c44346357\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") "
Sep 18 20:14:18.071671 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071495    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert\") pod \"94ba9025-4999-435c-83bb-ac8c44346357\" (UID: \"94ba9025-4999-435c-83bb-ac8c44346357\") "
Sep 18 20:14:18.071671 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071622    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94ba9025-4999-435c-83bb-ac8c44346357" (UID: "94ba9025-4999-435c-83bb-ac8c44346357"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:18.071671 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071624    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94ba9025-4999-435c-83bb-ac8c44346357" (UID: "94ba9025-4999-435c-83bb-ac8c44346357"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:18.071860 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071840    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94ba9025-4999-435c-83bb-ac8c44346357" (UID: "94ba9025-4999-435c-83bb-ac8c44346357"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:18.071912 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071849    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94ba9025-4999-435c-83bb-ac8c44346357-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:18.071912 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.071878    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:18.073477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.073454    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94ba9025-4999-435c-83bb-ac8c44346357" (UID: "94ba9025-4999-435c-83bb-ac8c44346357"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:18.073575 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.073492    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq" (OuterVolumeSpecName: "kube-api-access-gbbsq") pod "94ba9025-4999-435c-83bb-ac8c44346357" (UID: "94ba9025-4999-435c-83bb-ac8c44346357"). InnerVolumeSpecName "kube-api-access-gbbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:18.172505 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.172467    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94ba9025-4999-435c-83bb-ac8c44346357-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:18.172505 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.172496    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbsq\" (UniqueName: \"kubernetes.io/projected/94ba9025-4999-435c-83bb-ac8c44346357-kube-api-access-gbbsq\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:18.172505 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.172507    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94ba9025-4999-435c-83bb-ac8c44346357-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:18.559784 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.559744    2573 generic.go:356] "Generic (PLEG): container finished" podID="94ba9025-4999-435c-83bb-ac8c44346357" containerID="1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397" exitCode=0
Sep 18 20:14:18.559988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.559800    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"
Sep 18 20:14:18.559988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.559820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerDied","Data":"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"}
Sep 18 20:14:18.559988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.559886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8" event={"ID":"94ba9025-4999-435c-83bb-ac8c44346357","Type":"ContainerDied","Data":"777661cbb04b38da25c104be4e5a6de15ee63593e2ff9e582fecf58f5e64bcb2"}
Sep 18 20:14:18.559988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.559915    2573 scope.go:117] "RemoveContainer" containerID="267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"
Sep 18 20:14:18.569280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.569184    2573 scope.go:117] "RemoveContainer" containerID="1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"
Sep 18 20:14:18.580173 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.580151    2573 scope.go:117] "RemoveContainer" containerID="0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"
Sep 18 20:14:18.588542 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.588512    2573 scope.go:117] "RemoveContainer" containerID="267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"
Sep 18 20:14:18.588827 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:14:18.588802    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370\": container with ID starting with 267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370 not found: ID does not exist" containerID="267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"
Sep 18 20:14:18.588894 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.588838    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370"} err="failed to get container status \"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370\": rpc error: code = NotFound desc = could not find container \"267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370\": container with ID starting with 267e23b18e858ba61ffdf097278bb175323c6d7814249d59cdefec01745a5370 not found: ID does not exist"
Sep 18 20:14:18.588894 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.588861    2573 scope.go:117] "RemoveContainer" containerID="1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"
Sep 18 20:14:18.588894 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.588878    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:14:18.589149 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:14:18.589133    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397\": container with ID starting with 1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397 not found: ID does not exist" containerID="1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"
Sep 18 20:14:18.589194 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.589152    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397"} err="failed to get container status \"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397\": rpc error: code = NotFound desc = could not find container \"1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397\": container with ID starting with 1036f49c42adb48bbcc166bccf9eb3937b9270d9da9b7ac18dfbdd6df81e5397 not found: ID does not exist"
Sep 18 20:14:18.589194 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.589165    2573 scope.go:117] "RemoveContainer" containerID="0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"
Sep 18 20:14:18.589441 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:14:18.589420    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e\": container with ID starting with 0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e not found: ID does not exist" containerID="0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"
Sep 18 20:14:18.589497 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.589449    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e"} err="failed to get container status \"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e\": rpc error: code = NotFound desc = could not find container \"0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e\": container with ID starting with 0948245fc8f8c099e22704b874aee4e4f0145c05070235ffc5525d6c1d2a664e not found: ID does not exist"
Sep 18 20:14:18.596374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.596349    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ptfg8"]
Sep 18 20:14:18.632613 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632577    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:14:18.632913 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632897    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632914    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632925    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kube-rbac-proxy"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632931    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kube-rbac-proxy"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632946    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="storage-initializer"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632952    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="storage-initializer"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632959    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632964    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632974    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kube-rbac-proxy"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632980    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kube-rbac-proxy"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632990    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="storage-initializer"
Sep 18 20:14:18.633004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.632998    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="storage-initializer"
Sep 18 20:14:18.633589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.633047    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kserve-container"
Sep 18 20:14:18.633589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.633054    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kserve-container"
Sep 18 20:14:18.633589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.633065    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ba9025-4999-435c-83bb-ac8c44346357" containerName="kube-rbac-proxy"
Sep 18 20:14:18.633589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.633073    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b6eedbd-6da3-4d05-b3a0-b82deb741c2d" containerName="kube-rbac-proxy"
Sep 18 20:14:18.636060 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.636041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.638228 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.638193    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:14:18.638324 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.638192    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 20:14:18.644699 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.644679    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:14:18.676528 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.676500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.676648 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.676536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhng4\" (UniqueName: \"kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.676648 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.676563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.676648 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.676623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.676762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.676648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777292 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhng4\" (UniqueName: \"kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.777786 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.777758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.778059 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.778038    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.778134 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.778086    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.779856 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.779831    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.787781 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.787760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhng4\" (UniqueName: \"kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:18.946004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:18.945969    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:19.215955 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:19.215882    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:14:19.219643 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:14:19.219616    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f866180_9422_4dc8_9a3c_e9e360e349fd.slice/crio-bb330280f810ce80d095c6ef1bc79024ce1e363a29c342cdd3be21a2906e0b94 WatchSource:0}: Error finding container bb330280f810ce80d095c6ef1bc79024ce1e363a29c342cdd3be21a2906e0b94: Status 404 returned error can't find the container with id bb330280f810ce80d095c6ef1bc79024ce1e363a29c342cdd3be21a2906e0b94
Sep 18 20:14:19.565605 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:19.565496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerStarted","Data":"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"}
Sep 18 20:14:19.565766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:19.565610    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerStarted","Data":"bb330280f810ce80d095c6ef1bc79024ce1e363a29c342cdd3be21a2906e0b94"}
Sep 18 20:14:19.782681 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:19.782652    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ba9025-4999-435c-83bb-ac8c44346357" path="/var/lib/kubelet/pods/94ba9025-4999-435c-83bb-ac8c44346357/volumes"
Sep 18 20:14:21.340849 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:21.340803    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:14:23.585989 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:23.585951    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerID="0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a" exitCode=0
Sep 18 20:14:23.586482 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:23.586022    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerDied","Data":"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"}
Sep 18 20:14:24.592402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:24.592366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerStarted","Data":"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"}
Sep 18 20:14:24.592838 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:24.592453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerStarted","Data":"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"}
Sep 18 20:14:24.592838 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:24.592667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:24.592838 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:24.592808    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:24.612031 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:24.611979    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" podStartSLOduration=2.814640301 podStartE2EDuration="6.611967267s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.797326966s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:24.610664369 +0000 UTC m=+2173.128634008" watchObservedRunningTime="2026-09-18 20:14:24.611967267 +0000 UTC m=+2173.129936907"
Sep 18 20:14:30.602243 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:30.602195    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:14:31.340904 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:31.340858    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:14:33.780682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:33.780629    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 20:14:43.782833 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:43.782803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:14:47.830247 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:47.830211    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:14:47.830671 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:47.830452    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container" containerID="cri-o://8ec6ceab8b1e826983eb7d48e7f7cdbf768b5eaa5d72ef2a90bb7b7f76d6f8d4" gracePeriod=30
Sep 18 20:14:47.830671 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:47.830509    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kube-rbac-proxy" containerID="cri-o://084024667bd1b263dd38913a80921afc129d8c9af2d9e0074d8ea216be12dc9e" gracePeriod=30
Sep 18 20:14:48.702557 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:48.702519    2573 generic.go:356] "Generic (PLEG): container finished" podID="3070b071-f7e8-4abc-877f-02811d4fa750" containerID="084024667bd1b263dd38913a80921afc129d8c9af2d9e0074d8ea216be12dc9e" exitCode=2
Sep 18 20:14:48.702733 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:48.702581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerDied","Data":"084024667bd1b263dd38913a80921afc129d8c9af2d9e0074d8ea216be12dc9e"}
Sep 18 20:14:51.335851 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:51.335804    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 18 20:14:52.722036 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.722002    2573 generic.go:356] "Generic (PLEG): container finished" podID="3070b071-f7e8-4abc-877f-02811d4fa750" containerID="8ec6ceab8b1e826983eb7d48e7f7cdbf768b5eaa5d72ef2a90bb7b7f76d6f8d4" exitCode=0
Sep 18 20:14:52.722036 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.722028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerDied","Data":"8ec6ceab8b1e826983eb7d48e7f7cdbf768b5eaa5d72ef2a90bb7b7f76d6f8d4"}
Sep 18 20:14:52.776967 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.776941    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:14:52.780669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.780641    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls\") pod \"3070b071-f7e8-4abc-877f-02811d4fa750\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") "
Sep 18 20:14:52.780798 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.780696    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert\") pod \"3070b071-f7e8-4abc-877f-02811d4fa750\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") "
Sep 18 20:14:52.780798 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.780722    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location\") pod \"3070b071-f7e8-4abc-877f-02811d4fa750\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") "
Sep 18 20:14:52.780798 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.780759    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdmt\" (UniqueName: \"kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt\") pod \"3070b071-f7e8-4abc-877f-02811d4fa750\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") "
Sep 18 20:14:52.780966 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.780810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config\") pod \"3070b071-f7e8-4abc-877f-02811d4fa750\" (UID: \"3070b071-f7e8-4abc-877f-02811d4fa750\") "
Sep 18 20:14:52.781132 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.781053    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3070b071-f7e8-4abc-877f-02811d4fa750" (UID: "3070b071-f7e8-4abc-877f-02811d4fa750"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:14:52.781191 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.781156    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3070b071-f7e8-4abc-877f-02811d4fa750" (UID: "3070b071-f7e8-4abc-877f-02811d4fa750"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:52.781319 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.781297    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3070b071-f7e8-4abc-877f-02811d4fa750" (UID: "3070b071-f7e8-4abc-877f-02811d4fa750"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:14:52.782933 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.782905    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt" (OuterVolumeSpecName: "kube-api-access-pwdmt") pod "3070b071-f7e8-4abc-877f-02811d4fa750" (UID: "3070b071-f7e8-4abc-877f-02811d4fa750"). InnerVolumeSpecName "kube-api-access-pwdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:14:52.783055 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.782939    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3070b071-f7e8-4abc-877f-02811d4fa750" (UID: "3070b071-f7e8-4abc-877f-02811d4fa750"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:14:52.882008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.881979    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.882008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.882004    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3070b071-f7e8-4abc-877f-02811d4fa750-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.882008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.882014    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3070b071-f7e8-4abc-877f-02811d4fa750-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.882261 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.882022    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3070b071-f7e8-4abc-877f-02811d4fa750-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:52.882261 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:52.882032    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdmt\" (UniqueName: \"kubernetes.io/projected/3070b071-f7e8-4abc-877f-02811d4fa750-kube-api-access-pwdmt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:14:53.728627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.728595    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"
Sep 18 20:14:53.728627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.728613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd" event={"ID":"3070b071-f7e8-4abc-877f-02811d4fa750","Type":"ContainerDied","Data":"673ac9d4b7660f968f72e3265b92a9bb6d64c64557eabf6bcf7d655a331b6056"}
Sep 18 20:14:53.729132 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.728697    2573 scope.go:117] "RemoveContainer" containerID="084024667bd1b263dd38913a80921afc129d8c9af2d9e0074d8ea216be12dc9e"
Sep 18 20:14:53.738013 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.737991    2573 scope.go:117] "RemoveContainer" containerID="8ec6ceab8b1e826983eb7d48e7f7cdbf768b5eaa5d72ef2a90bb7b7f76d6f8d4"
Sep 18 20:14:53.745472 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.745455    2573 scope.go:117] "RemoveContainer" containerID="9b571c918bd31544cace1ab49a8065e1643e07224f0875ff61ed168cdff5af4b"
Sep 18 20:14:53.750221 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.750186    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:14:53.752766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.752744    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-lvdqd"]
Sep 18 20:14:53.781939 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:14:53.781911    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" path="/var/lib/kubelet/pods/3070b071-f7e8-4abc-877f-02811d4fa750/volumes"
Sep 18 20:15:00.606576 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:00.606537    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:15:08.674399 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:08.674367    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:15:08.675105 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:08.674616    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kserve-container" containerID="cri-o://5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f" gracePeriod=30
Sep 18 20:15:08.675105 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:08.674668    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy" containerID="cri-o://5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4" gracePeriod=30
Sep 18 20:15:08.794346 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:08.794314    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerID="5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4" exitCode=2
Sep 18 20:15:08.794495 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:08.794388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerDied","Data":"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"}
Sep 18 20:15:10.597562 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:10.597515    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 20:15:13.843526 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843488    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843825    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kube-rbac-proxy"
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843837    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kube-rbac-proxy"
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843862    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container"
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843870    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container"
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843883    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="storage-initializer"
Sep 18 20:15:13.843915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="storage-initializer"
Sep 18 20:15:13.844114 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843935    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kube-rbac-proxy"
Sep 18 20:15:13.844114 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.843948    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3070b071-f7e8-4abc-877f-02811d4fa750" containerName="kserve-container"
Sep 18 20:15:13.846510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.846487    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:13.848781 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.848756    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 20:15:13.848895 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.848761    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 20:15:13.857710 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.857683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:15:13.960145 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.960111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8qd\" (UniqueName: \"kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:13.960145 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.960149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:13.960396 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.960184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:13.960396 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.960308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:13.960396 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:13.960340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.061529 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.061489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.061529 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.061528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.061812 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.061549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8qd\" (UniqueName: \"kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.061812 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.061573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.061812 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.061605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.062036 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.062011    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.062348 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.062319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.062469 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.062409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.064290 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.064264    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.069371 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.069347    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8qd\" (UniqueName: \"kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xtj59\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.157957 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.157856    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:14.432352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.432189    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:15:14.435114 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:15:14.435069    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33e2276_8f82_4ac8_9ee7_afef8d5da87c.slice/crio-0d2ff5fecac9aaca84f09b34b25677f2499af20de2ca1628d0f8fb8848d97094 WatchSource:0}: Error finding container 0d2ff5fecac9aaca84f09b34b25677f2499af20de2ca1628d0f8fb8848d97094: Status 404 returned error can't find the container with id 0d2ff5fecac9aaca84f09b34b25677f2499af20de2ca1628d0f8fb8848d97094
Sep 18 20:15:14.819977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.819892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerStarted","Data":"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"}
Sep 18 20:15:14.819977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:14.819950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerStarted","Data":"0d2ff5fecac9aaca84f09b34b25677f2499af20de2ca1628d0f8fb8848d97094"}
Sep 18 20:15:15.597693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.597659    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 20:15:15.719668 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.719645    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:15:15.776224 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776156    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert\") pod \"2f866180-9422-4dc8-9a3c-e9e360e349fd\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") "
Sep 18 20:15:15.776402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776240    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhng4\" (UniqueName: \"kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4\") pod \"2f866180-9422-4dc8-9a3c-e9e360e349fd\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") "
Sep 18 20:15:15.776402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776264    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config\") pod \"2f866180-9422-4dc8-9a3c-e9e360e349fd\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") "
Sep 18 20:15:15.776402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776364    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location\") pod \"2f866180-9422-4dc8-9a3c-e9e360e349fd\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") "
Sep 18 20:15:15.776519 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776441    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls\") pod \"2f866180-9422-4dc8-9a3c-e9e360e349fd\" (UID: \"2f866180-9422-4dc8-9a3c-e9e360e349fd\") "
Sep 18 20:15:15.776658 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776636    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f866180-9422-4dc8-9a3c-e9e360e349fd" (UID: "2f866180-9422-4dc8-9a3c-e9e360e349fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:15.776706 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776648    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f866180-9422-4dc8-9a3c-e9e360e349fd" (UID: "2f866180-9422-4dc8-9a3c-e9e360e349fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:15:15.776706 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776665    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f866180-9422-4dc8-9a3c-e9e360e349fd" (UID: "2f866180-9422-4dc8-9a3c-e9e360e349fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:15:15.776799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776737    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:15:15.776799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776754    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:15:15.776799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.776764    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f866180-9422-4dc8-9a3c-e9e360e349fd-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:15:15.778556 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.778524    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f866180-9422-4dc8-9a3c-e9e360e349fd" (UID: "2f866180-9422-4dc8-9a3c-e9e360e349fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:15:15.778670 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.778621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4" (OuterVolumeSpecName: "kube-api-access-lhng4") pod "2f866180-9422-4dc8-9a3c-e9e360e349fd" (UID: "2f866180-9422-4dc8-9a3c-e9e360e349fd"). InnerVolumeSpecName "kube-api-access-lhng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:15:15.825747 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.825712    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerID="5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f" exitCode=0
Sep 18 20:15:15.825903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.825793    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"
Sep 18 20:15:15.825903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.825802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerDied","Data":"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"}
Sep 18 20:15:15.825903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.825874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp" event={"ID":"2f866180-9422-4dc8-9a3c-e9e360e349fd","Type":"ContainerDied","Data":"bb330280f810ce80d095c6ef1bc79024ce1e363a29c342cdd3be21a2906e0b94"}
Sep 18 20:15:15.825903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.825903    2573 scope.go:117] "RemoveContainer" containerID="5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"
Sep 18 20:15:15.834664 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.834645    2573 scope.go:117] "RemoveContainer" containerID="5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"
Sep 18 20:15:15.841349 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.841324    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:15:15.842660 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.842643    2573 scope.go:117] "RemoveContainer" containerID="0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"
Sep 18 20:15:15.845370 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.845350    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-n9qlp"]
Sep 18 20:15:15.852362 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.852344    2573 scope.go:117] "RemoveContainer" containerID="5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"
Sep 18 20:15:15.852839 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:15:15.852820    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4\": container with ID starting with 5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4 not found: ID does not exist" containerID="5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"
Sep 18 20:15:15.852909 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.852852    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4"} err="failed to get container status \"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4\": rpc error: code = NotFound desc = could not find container \"5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4\": container with ID starting with 5936d32a34dee1a3b6a2d1c0e79e754116596e42d41f6b8a866b24509eacc8a4 not found: ID does not exist"
Sep 18 20:15:15.852909 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.852879    2573 scope.go:117] "RemoveContainer" containerID="5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"
Sep 18 20:15:15.853189 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:15:15.853172    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f\": container with ID starting with 5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f not found: ID does not exist" containerID="5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"
Sep 18 20:15:15.853265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.853195    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f"} err="failed to get container status \"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f\": rpc error: code = NotFound desc = could not find container \"5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f\": container with ID starting with 5a6578cac165a0b74b23188edf5f949ade085922ce0a7f6c2e8fc66a5522997f not found: ID does not exist"
Sep 18 20:15:15.853265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.853263    2573 scope.go:117] "RemoveContainer" containerID="0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"
Sep 18 20:15:15.853537 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:15:15.853518    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a\": container with ID starting with 0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a not found: ID does not exist" containerID="0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"
Sep 18 20:15:15.853593 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.853546    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a"} err="failed to get container status \"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a\": rpc error: code = NotFound desc = could not find container \"0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a\": container with ID starting with 0b93f977e8707845fd75bea28c842d9eed802a1cb9f137ad3627eeeea369aa7a not found: ID does not exist"
Sep 18 20:15:15.878143 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.878112    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhng4\" (UniqueName: \"kubernetes.io/projected/2f866180-9422-4dc8-9a3c-e9e360e349fd-kube-api-access-lhng4\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:15:15.878332 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:15.878307    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f866180-9422-4dc8-9a3c-e9e360e349fd-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:15:17.782023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:17.781988    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" path="/var/lib/kubelet/pods/2f866180-9422-4dc8-9a3c-e9e360e349fd/volumes"
Sep 18 20:15:18.839687 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:18.839655    2573 generic.go:356] "Generic (PLEG): container finished" podID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerID="b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163" exitCode=0
Sep 18 20:15:18.840077 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:18.839707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerDied","Data":"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"}
Sep 18 20:15:19.846092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:19.846056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerStarted","Data":"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"}
Sep 18 20:15:19.846514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:19.846120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerStarted","Data":"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"}
Sep 18 20:15:19.846514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:19.846461    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:19.846514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:19.846516    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:19.866663 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:19.866607    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" podStartSLOduration=3.182252122 podStartE2EDuration="6.866590613s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684338491s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:15:19.864330159 +0000 UTC m=+2228.382299799" watchObservedRunningTime="2026-09-18 20:15:19.866590613 +0000 UTC m=+2228.384560253"
Sep 18 20:15:25.856301 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:25.856216    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:15:55.859983 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:15:55.859952    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:16:03.944380 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:03.944348    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:16:03.944777 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:03.944591    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kserve-container" containerID="cri-o://56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56" gracePeriod=30
Sep 18 20:16:03.944777 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:03.944641    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kube-rbac-proxy" containerID="cri-o://c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da" gracePeriod=30
Sep 18 20:16:05.031558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:05.031522    2573 generic.go:356] "Generic (PLEG): container finished" podID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerID="c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da" exitCode=2
Sep 18 20:16:05.032000 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:05.031594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerDied","Data":"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"}
Sep 18 20:16:05.851170 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:05.851125    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 18 20:16:09.118145 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118107    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118448    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="storage-initializer"
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118460    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="storage-initializer"
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kserve-container"
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118477    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kserve-container"
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118496    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy"
Sep 18 20:16:09.118552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118502    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy"
Sep 18 20:16:09.118746 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118548    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kube-rbac-proxy"
Sep 18 20:16:09.118746 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.118575    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f866180-9422-4dc8-9a3c-e9e360e349fd" containerName="kserve-container"
Sep 18 20:16:09.121821 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.121802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.124043 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.124023    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:16:09.124153 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.124025    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 18 20:16:09.132434 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.132407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:16:09.216396 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.216363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.216601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.216408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.216601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.216477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.216601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.216555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ttn\" (UniqueName: \"kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.216754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.216638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317456 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ttn\" (UniqueName: \"kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.317994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.317974    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.318245 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.318194    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.318315 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.318281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.320109 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.320088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.325381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.325357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ttn\" (UniqueName: \"kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-h8hwc\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.432501 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.432409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:09.704291 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:09.704263    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:16:09.706838 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:16:09.706805    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a364d3c_5d1a_4ff6_9991_92e1490e1daf.slice/crio-2604a1cedb64d3dd83e87eaea9d8acd86e82d74e3a188dfa194014dfc43be6f5 WatchSource:0}: Error finding container 2604a1cedb64d3dd83e87eaea9d8acd86e82d74e3a188dfa194014dfc43be6f5: Status 404 returned error can't find the container with id 2604a1cedb64d3dd83e87eaea9d8acd86e82d74e3a188dfa194014dfc43be6f5
Sep 18 20:16:10.056001 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.055900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerStarted","Data":"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"}
Sep 18 20:16:10.056001 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.055969    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerStarted","Data":"2604a1cedb64d3dd83e87eaea9d8acd86e82d74e3a188dfa194014dfc43be6f5"}
Sep 18 20:16:10.691724 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.691692    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:16:10.831328 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831232    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr8qd\" (UniqueName: \"kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd\") pod \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") "
Sep 18 20:16:10.831328 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831274    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location\") pod \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") "
Sep 18 20:16:10.831328 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831297    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert\") pod \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") "
Sep 18 20:16:10.831328 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831327    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls\") pod \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") "
Sep 18 20:16:10.831688 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831383    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config\") pod \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\" (UID: \"a33e2276-8f82-4ac8-9ee7-afef8d5da87c\") "
Sep 18 20:16:10.831688 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831641    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a33e2276-8f82-4ac8-9ee7-afef8d5da87c" (UID: "a33e2276-8f82-4ac8-9ee7-afef8d5da87c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:10.831788 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831699    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a33e2276-8f82-4ac8-9ee7-afef8d5da87c" (UID: "a33e2276-8f82-4ac8-9ee7-afef8d5da87c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:10.831826 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.831797    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a33e2276-8f82-4ac8-9ee7-afef8d5da87c" (UID: "a33e2276-8f82-4ac8-9ee7-afef8d5da87c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:10.833381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.833361    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd" (OuterVolumeSpecName: "kube-api-access-cr8qd") pod "a33e2276-8f82-4ac8-9ee7-afef8d5da87c" (UID: "a33e2276-8f82-4ac8-9ee7-afef8d5da87c"). InnerVolumeSpecName "kube-api-access-cr8qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:10.833481 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.833461    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a33e2276-8f82-4ac8-9ee7-afef8d5da87c" (UID: "a33e2276-8f82-4ac8-9ee7-afef8d5da87c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:10.932925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.932889    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:16:10.932925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.932917    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8qd\" (UniqueName: \"kubernetes.io/projected/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kube-api-access-cr8qd\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:16:10.932925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.932926    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:16:10.932925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.932936    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:16:10.933234 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:10.932946    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a33e2276-8f82-4ac8-9ee7-afef8d5da87c-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:16:11.061932 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.061895    2573 generic.go:356] "Generic (PLEG): container finished" podID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerID="56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56" exitCode=0
Sep 18 20:16:11.062135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.061968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"
Sep 18 20:16:11.062135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.061987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerDied","Data":"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"}
Sep 18 20:16:11.062135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.062045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59" event={"ID":"a33e2276-8f82-4ac8-9ee7-afef8d5da87c","Type":"ContainerDied","Data":"0d2ff5fecac9aaca84f09b34b25677f2499af20de2ca1628d0f8fb8848d97094"}
Sep 18 20:16:11.062135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.062071    2573 scope.go:117] "RemoveContainer" containerID="c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"
Sep 18 20:16:11.071268 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.071249    2573 scope.go:117] "RemoveContainer" containerID="56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"
Sep 18 20:16:11.078811 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.078785    2573 scope.go:117] "RemoveContainer" containerID="b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"
Sep 18 20:16:11.084216 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.084150    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:16:11.086979 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.086961    2573 scope.go:117] "RemoveContainer" containerID="c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"
Sep 18 20:16:11.087345 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:16:11.087313    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da\": container with ID starting with c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da not found: ID does not exist" containerID="c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"
Sep 18 20:16:11.087444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087353    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da"} err="failed to get container status \"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da\": rpc error: code = NotFound desc = could not find container \"c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da\": container with ID starting with c69216b85efa4edb0ddc0d8e31874b0137d9e04f16c81262b524f0b801c8e6da not found: ID does not exist"
Sep 18 20:16:11.087444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087372    2573 scope.go:117] "RemoveContainer" containerID="56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"
Sep 18 20:16:11.087652 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:16:11.087634    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56\": container with ID starting with 56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56 not found: ID does not exist" containerID="56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"
Sep 18 20:16:11.087717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087661    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56"} err="failed to get container status \"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56\": rpc error: code = NotFound desc = could not find container \"56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56\": container with ID starting with 56bee971f3fcc968e95c336ac0449e4771ba0a39b651b5eaf46d4934880a5d56 not found: ID does not exist"
Sep 18 20:16:11.087717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087685    2573 scope.go:117] "RemoveContainer" containerID="b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"
Sep 18 20:16:11.087918 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087902    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xtj59"]
Sep 18 20:16:11.087972 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:16:11.087953    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163\": container with ID starting with b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163 not found: ID does not exist" containerID="b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"
Sep 18 20:16:11.088012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.087979    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163"} err="failed to get container status \"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163\": rpc error: code = NotFound desc = could not find container \"b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163\": container with ID starting with b446481634e066fc8d195414a7bf33c62a183b86851c7244ef5f2d2a5b822163 not found: ID does not exist"
Sep 18 20:16:11.782800 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:11.782770    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" path="/var/lib/kubelet/pods/a33e2276-8f82-4ac8-9ee7-afef8d5da87c/volumes"
Sep 18 20:16:14.076413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:14.076375    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerID="1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02" exitCode=0
Sep 18 20:16:14.076809 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:14.076436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerDied","Data":"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"}
Sep 18 20:16:15.082958 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.082925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerStarted","Data":"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"}
Sep 18 20:16:15.083386 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.082986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerStarted","Data":"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"}
Sep 18 20:16:15.083386 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.083239    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:15.083386 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.083292    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:15.084811 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.084783    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:16:15.103693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:15.103595    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podStartSLOduration=2.351644738 podStartE2EDuration="6.103579685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.751934947s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:16:15.101884526 +0000 UTC m=+2283.619854168" watchObservedRunningTime="2026-09-18 20:16:15.103579685 +0000 UTC m=+2283.621549324"
Sep 18 20:16:16.086933 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:16.086892    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:16:21.091861 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:21.091828    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:16:21.092512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:21.092483    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:16:31.093068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:31.093026    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:16:41.093429 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:41.093386    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:16:51.093369 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:16:51.093330    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:17:01.092548 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:01.092471    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:17:03.769607 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769573    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769901    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kserve-container"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769912    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kserve-container"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769935    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="storage-initializer"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769941    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="storage-initializer"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769950    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kube-rbac-proxy"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.769956    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kube-rbac-proxy"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.770005    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kserve-container"
Sep 18 20:17:03.770093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.770015    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33e2276-8f82-4ac8-9ee7-afef8d5da87c" containerName="kube-rbac-proxy"
Sep 18 20:17:03.773304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.773279    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.775512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.775487    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 18 20:17:03.775619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.775586    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:17:03.783846 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.783823    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:17:03.887420 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.887374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9cnt\" (UniqueName: \"kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.887764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.887737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.887892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.887869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.887982 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.887905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.888065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.888042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989283 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9cnt\" (UniqueName: \"kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989283 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989364    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.989802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.989779    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.990075 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.990059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.990123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.990105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.991892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.991874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:03.996876 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:03.996853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9cnt\" (UniqueName: \"kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:04.085783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:04.085702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:04.356392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:04.356364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:17:04.358589 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:17:04.358561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19629757_fe01_454f_8c69_43c69a7bad31.slice/crio-aa430bd18fd1a9e1994d90f5b093dc34e6f91a1acf6736fdd60dc96f2abb3dfb WatchSource:0}: Error finding container aa430bd18fd1a9e1994d90f5b093dc34e6f91a1acf6736fdd60dc96f2abb3dfb: Status 404 returned error can't find the container with id aa430bd18fd1a9e1994d90f5b093dc34e6f91a1acf6736fdd60dc96f2abb3dfb
Sep 18 20:17:05.281394 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:05.281360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerStarted","Data":"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"}
Sep 18 20:17:05.281771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:05.281412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerStarted","Data":"aa430bd18fd1a9e1994d90f5b093dc34e6f91a1acf6736fdd60dc96f2abb3dfb"}
Sep 18 20:17:08.299288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:08.299252    2573 generic.go:356] "Generic (PLEG): container finished" podID="19629757-fe01-454f-8c69-43c69a7bad31" containerID="dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e" exitCode=0
Sep 18 20:17:08.299763 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:08.299328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerDied","Data":"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"}
Sep 18 20:17:08.300686 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:08.300670    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:17:11.093340 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:11.093303    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:17:15.335329 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:15.335246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerStarted","Data":"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"}
Sep 18 20:17:15.335329 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:15.335318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerStarted","Data":"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"}
Sep 18 20:17:15.335714 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:15.335506    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:15.353916 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:15.353849    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podStartSLOduration=2.052418015 podStartE2EDuration="12.353833527s" totalImagesPullingTime="6.662211633s" totalInitContainerRuntime="3.639203879s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:15.353051719 +0000 UTC m=+2343.871021362" watchObservedRunningTime="2026-09-18 20:17:15.353833527 +0000 UTC m=+2343.871803166"
Sep 18 20:17:16.339438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:16.339404    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:16.340615 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:16.340590    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:17.345316 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:17.345278    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:21.092656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:21.092613    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:17:22.349421 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:22.349396    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:17:22.349917 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:22.349890    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:31.093377 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:31.093345    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:17:32.350176 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:32.350136    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:39.835541 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:39.835510    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:17:39.836149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:39.835735    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" containerID="cri-o://231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2" gracePeriod=30
Sep 18 20:17:39.836149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:39.835774    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kube-rbac-proxy" containerID="cri-o://91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e" gracePeriod=30
Sep 18 20:17:40.448565 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:40.448531    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerID="91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e" exitCode=2
Sep 18 20:17:40.448723 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:40.448573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerDied","Data":"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"}
Sep 18 20:17:41.087691 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:41.087650    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 18 20:17:41.092887 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:41.092859    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 18 20:17:42.350254 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:42.350220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:43.876287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.876263    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:17:43.915561 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915528    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55ttn\" (UniqueName: \"kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn\") pod \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") "
Sep 18 20:17:43.915714 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls\") pod \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") "
Sep 18 20:17:43.915714 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915676    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location\") pod \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") "
Sep 18 20:17:43.915838 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915720    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config\") pod \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") "
Sep 18 20:17:43.915838 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915751    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert\") pod \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\" (UID: \"0a364d3c-5d1a-4ff6-9991-92e1490e1daf\") "
Sep 18 20:17:43.916146 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.915994    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a364d3c-5d1a-4ff6-9991-92e1490e1daf" (UID: "0a364d3c-5d1a-4ff6-9991-92e1490e1daf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:43.916242 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.916215    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a364d3c-5d1a-4ff6-9991-92e1490e1daf" (UID: "0a364d3c-5d1a-4ff6-9991-92e1490e1daf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:43.916353 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.916332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a364d3c-5d1a-4ff6-9991-92e1490e1daf" (UID: "0a364d3c-5d1a-4ff6-9991-92e1490e1daf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:17:43.917703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.917682    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn" (OuterVolumeSpecName: "kube-api-access-55ttn") pod "0a364d3c-5d1a-4ff6-9991-92e1490e1daf" (UID: "0a364d3c-5d1a-4ff6-9991-92e1490e1daf"). InnerVolumeSpecName "kube-api-access-55ttn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:43.917791 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:43.917706    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a364d3c-5d1a-4ff6-9991-92e1490e1daf" (UID: "0a364d3c-5d1a-4ff6-9991-92e1490e1daf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:44.016923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.016843    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:17:44.016923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.016871    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:17:44.016923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.016880    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:17:44.016923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.016891    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:17:44.016923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.016900    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ttn\" (UniqueName: \"kubernetes.io/projected/0a364d3c-5d1a-4ff6-9991-92e1490e1daf-kube-api-access-55ttn\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:17:44.465704 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.465671    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerID="231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2" exitCode=0
Sep 18 20:17:44.465867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.465726    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"
Sep 18 20:17:44.465867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.465757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerDied","Data":"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"}
Sep 18 20:17:44.465867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.465817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc" event={"ID":"0a364d3c-5d1a-4ff6-9991-92e1490e1daf","Type":"ContainerDied","Data":"2604a1cedb64d3dd83e87eaea9d8acd86e82d74e3a188dfa194014dfc43be6f5"}
Sep 18 20:17:44.465867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.465845    2573 scope.go:117] "RemoveContainer" containerID="91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"
Sep 18 20:17:44.474617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.474596    2573 scope.go:117] "RemoveContainer" containerID="231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"
Sep 18 20:17:44.481682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.481663    2573 scope.go:117] "RemoveContainer" containerID="1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"
Sep 18 20:17:44.489111 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489087    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:17:44.489349 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489334    2573 scope.go:117] "RemoveContainer" containerID="91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"
Sep 18 20:17:44.489621 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:17:44.489606    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e\": container with ID starting with 91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e not found: ID does not exist" containerID="91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"
Sep 18 20:17:44.489668 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489630    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e"} err="failed to get container status \"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e\": rpc error: code = NotFound desc = could not find container \"91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e\": container with ID starting with 91032d1c27ebf974fe8c3d4ca7215d918a84c3ded16ed645285016789d91fc1e not found: ID does not exist"
Sep 18 20:17:44.489668 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489645    2573 scope.go:117] "RemoveContainer" containerID="231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"
Sep 18 20:17:44.489905 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:17:44.489887    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2\": container with ID starting with 231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2 not found: ID does not exist" containerID="231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"
Sep 18 20:17:44.489956 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489910    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2"} err="failed to get container status \"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2\": rpc error: code = NotFound desc = could not find container \"231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2\": container with ID starting with 231d865f474fd452d6739880b818702aa4be582f9873a22b1e70edea529274e2 not found: ID does not exist"
Sep 18 20:17:44.489956 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.489925    2573 scope.go:117] "RemoveContainer" containerID="1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"
Sep 18 20:17:44.490150 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:17:44.490133    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02\": container with ID starting with 1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02 not found: ID does not exist" containerID="1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"
Sep 18 20:17:44.490190 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.490154    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02"} err="failed to get container status \"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02\": rpc error: code = NotFound desc = could not find container \"1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02\": container with ID starting with 1d76348efbb14b83f48394d376d66fbf7aa18a0b2af2f997e6b9f89a36fcdb02 not found: ID does not exist"
Sep 18 20:17:44.493530 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:44.493512    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-h8hwc"]
Sep 18 20:17:45.027506 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.027474    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:17:45.027972 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.027954    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="storage-initializer"
Sep 18 20:17:45.028017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.027976    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="storage-initializer"
Sep 18 20:17:45.028017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028004    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container"
Sep 18 20:17:45.028017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028014    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container"
Sep 18 20:17:45.028116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028028    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kube-rbac-proxy"
Sep 18 20:17:45.028116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028039    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kube-rbac-proxy"
Sep 18 20:17:45.028177 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028116    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kube-rbac-proxy"
Sep 18 20:17:45.028177 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.028126    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" containerName="kserve-container"
Sep 18 20:17:45.031627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.031605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.033850 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.033832    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 18 20:17:45.033947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.033906    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:17:45.041024 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.041001    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:17:45.125041 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.125014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.125160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.125048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mpd\" (UniqueName: \"kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.125160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.125070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.125160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.125087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.125160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.125148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226282 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mpd\" (UniqueName: \"kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.226696 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:17:45.226607    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 18 20:17:45.226696 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:17:45.226682    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls podName:c0312abb-9798-49b1-9025-06d9f47d761e nodeName:}" failed. No retries permitted until 2026-09-18 20:17:45.726659743 +0000 UTC m=+2374.244629380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 18 20:17:45.226813 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.226712    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.227167 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.227145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.227229 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.227145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.234938 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.234915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mpd\" (UniqueName: \"kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.730599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.730566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.733073 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.733039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:45.782362 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.782331    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a364d3c-5d1a-4ff6-9991-92e1490e1daf" path="/var/lib/kubelet/pods/0a364d3c-5d1a-4ff6-9991-92e1490e1daf/volumes"
Sep 18 20:17:45.943791 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:45.943751    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:46.213822 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:46.213788    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:17:46.216546 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:17:46.216521    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0312abb_9798_49b1_9025_06d9f47d761e.slice/crio-e71e8aef812643b15dcd194dd98104f5750dee16ed509f304f8ee1ce125ddca7 WatchSource:0}: Error finding container e71e8aef812643b15dcd194dd98104f5750dee16ed509f304f8ee1ce125ddca7: Status 404 returned error can't find the container with id e71e8aef812643b15dcd194dd98104f5750dee16ed509f304f8ee1ce125ddca7
Sep 18 20:17:46.482335 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:46.482299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerStarted","Data":"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"}
Sep 18 20:17:46.482502 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:46.482362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerStarted","Data":"e71e8aef812643b15dcd194dd98104f5750dee16ed509f304f8ee1ce125ddca7"}
Sep 18 20:17:50.500055 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:50.500020    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0312abb-9798-49b1-9025-06d9f47d761e" containerID="851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3" exitCode=0
Sep 18 20:17:50.500504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:50.500091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerDied","Data":"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"}
Sep 18 20:17:51.506700 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:51.506665    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerStarted","Data":"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"}
Sep 18 20:17:51.507067 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:51.506762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerStarted","Data":"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"}
Sep 18 20:17:51.507123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:51.507060    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:51.525048 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:51.525002    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podStartSLOduration=2.477667919 podStartE2EDuration="6.524990085s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.047322166s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:51.523904721 +0000 UTC m=+2380.041874363" watchObservedRunningTime="2026-09-18 20:17:51.524990085 +0000 UTC m=+2380.042959724"
Sep 18 20:17:52.350835 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:52.350802    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:17:52.511128 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:52.511100    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:17:58.520316 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:17:58.520280    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:18:02.349986 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:02.349948    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:12.349907 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:12.349869    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:22.350264 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:22.350227    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:28.527056 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:28.526964    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 20:18:32.349816 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:32.349775    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:36.777572 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:36.777532    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:38.523047 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:38.523020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:18:45.101145 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:45.101112    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:18:45.101681 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:45.101365    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container" containerID="cri-o://86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704" gracePeriod=30
Sep 18 20:18:45.101681 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:45.101415    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kube-rbac-proxy" containerID="cri-o://0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1" gracePeriod=30
Sep 18 20:18:45.732007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:45.731967    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0312abb-9798-49b1-9025-06d9f47d761e" containerID="0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1" exitCode=2
Sep 18 20:18:45.732175 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:45.732015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerDied","Data":"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"}
Sep 18 20:18:46.778435 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:46.778403    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:18:48.516014 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:48.515966    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 18 20:18:48.520791 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:48.520763    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 20:18:52.240495 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.240442    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:18:52.385749 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.385716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location\") pod \"c0312abb-9798-49b1-9025-06d9f47d761e\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") "
Sep 18 20:18:52.385949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.385767    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5mpd\" (UniqueName: \"kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd\") pod \"c0312abb-9798-49b1-9025-06d9f47d761e\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") "
Sep 18 20:18:52.385949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.385811    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") pod \"c0312abb-9798-49b1-9025-06d9f47d761e\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") "
Sep 18 20:18:52.385949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.385863    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert\") pod \"c0312abb-9798-49b1-9025-06d9f47d761e\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") "
Sep 18 20:18:52.385949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.385893    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config\") pod \"c0312abb-9798-49b1-9025-06d9f47d761e\" (UID: \"c0312abb-9798-49b1-9025-06d9f47d761e\") "
Sep 18 20:18:52.386228 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.386089    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0312abb-9798-49b1-9025-06d9f47d761e" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:18:52.386275 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.386222    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0312abb-9798-49b1-9025-06d9f47d761e" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:18:52.386307 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.386286    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0312abb-9798-49b1-9025-06d9f47d761e" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:18:52.387969 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.387944    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0312abb-9798-49b1-9025-06d9f47d761e" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:18:52.388046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.387990    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd" (OuterVolumeSpecName: "kube-api-access-d5mpd") pod "c0312abb-9798-49b1-9025-06d9f47d761e" (UID: "c0312abb-9798-49b1-9025-06d9f47d761e"). InnerVolumeSpecName "kube-api-access-d5mpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:18:52.487325 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.487292    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0312abb-9798-49b1-9025-06d9f47d761e-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:18:52.487325 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.487322    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mpd\" (UniqueName: \"kubernetes.io/projected/c0312abb-9798-49b1-9025-06d9f47d761e-kube-api-access-d5mpd\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:18:52.487525 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.487334    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0312abb-9798-49b1-9025-06d9f47d761e-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:18:52.487525 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.487347    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:18:52.487525 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.487358    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0312abb-9798-49b1-9025-06d9f47d761e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:18:52.763029 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.762994    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0312abb-9798-49b1-9025-06d9f47d761e" containerID="86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704" exitCode=0
Sep 18 20:18:52.763241 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.763039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerDied","Data":"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"}
Sep 18 20:18:52.763241 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.763045    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"
Sep 18 20:18:52.763241 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.763098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8" event={"ID":"c0312abb-9798-49b1-9025-06d9f47d761e","Type":"ContainerDied","Data":"e71e8aef812643b15dcd194dd98104f5750dee16ed509f304f8ee1ce125ddca7"}
Sep 18 20:18:52.763241 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.763134    2573 scope.go:117] "RemoveContainer" containerID="0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"
Sep 18 20:18:52.771659 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.771643    2573 scope.go:117] "RemoveContainer" containerID="86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"
Sep 18 20:18:52.778846 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.778830    2573 scope.go:117] "RemoveContainer" containerID="851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"
Sep 18 20:18:52.784101 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.784079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:18:52.786691 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.786621    2573 scope.go:117] "RemoveContainer" containerID="0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"
Sep 18 20:18:52.787005 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:18:52.786982    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1\": container with ID starting with 0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1 not found: ID does not exist" containerID="0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"
Sep 18 20:18:52.787081 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.787014    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1"} err="failed to get container status \"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1\": rpc error: code = NotFound desc = could not find container \"0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1\": container with ID starting with 0752652c045750ef882b301c355334dbcaa929ff0ee229a6bd1fb0473cad31e1 not found: ID does not exist"
Sep 18 20:18:52.787081 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.787031    2573 scope.go:117] "RemoveContainer" containerID="86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"
Sep 18 20:18:52.787345 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:18:52.787319    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704\": container with ID starting with 86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704 not found: ID does not exist" containerID="86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"
Sep 18 20:18:52.787405 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.787352    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704"} err="failed to get container status \"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704\": rpc error: code = NotFound desc = could not find container \"86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704\": container with ID starting with 86b74ef62871f08ca492c19c717d785fba599ae5359edf3d51cc555808994704 not found: ID does not exist"
Sep 18 20:18:52.787405 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.787367    2573 scope.go:117] "RemoveContainer" containerID="851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"
Sep 18 20:18:52.787665 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:18:52.787650    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3\": container with ID starting with 851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3 not found: ID does not exist" containerID="851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"
Sep 18 20:18:52.787725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.787668    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3"} err="failed to get container status \"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3\": rpc error: code = NotFound desc = could not find container \"851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3\": container with ID starting with 851a4ad1ce009db1fe08f0eb9ba6753307c4fec8a79a456e320f8a65ba9f7ee3 not found: ID does not exist"
Sep 18 20:18:52.788418 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:52.788400    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-584p8"]
Sep 18 20:18:53.781487 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:53.781457    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" path="/var/lib/kubelet/pods/c0312abb-9798-49b1-9025-06d9f47d761e/volumes"
Sep 18 20:18:55.125834 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:55.125800    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:18:55.126639 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:55.126578    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" containerID="cri-o://49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb" gracePeriod=30
Sep 18 20:18:55.126761 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:55.126628    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kube-rbac-proxy" containerID="cri-o://977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a" gracePeriod=30
Sep 18 20:18:55.779540 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:55.779507    2573 generic.go:356] "Generic (PLEG): container finished" podID="19629757-fe01-454f-8c69-43c69a7bad31" containerID="977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a" exitCode=2
Sep 18 20:18:55.782524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:55.782503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerDied","Data":"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"}
Sep 18 20:18:56.777704 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:56.777666    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 20:18:57.346274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:18:57.346235    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 18 20:19:00.276388 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276153    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276711    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kube-rbac-proxy"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276734    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kube-rbac-proxy"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276757    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276768    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="storage-initializer"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276792    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="storage-initializer"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276860    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kube-rbac-proxy"
Sep 18 20:19:00.276923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.276874    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0312abb-9798-49b1-9025-06d9f47d761e" containerName="kserve-container"
Sep 18 20:19:00.280390 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.280367    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.282565 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.282542    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 18 20:19:00.282565 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.282557    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:19:00.289043 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.289020    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:00.373836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.373813    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:19:00.454486 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.454455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.454612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.454488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.454612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.454511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.454612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.454563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.454721 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.454611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvg7\" (UniqueName: \"kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.555584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.555516    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9cnt\" (UniqueName: \"kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt\") pod \"19629757-fe01-454f-8c69-43c69a7bad31\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") "
Sep 18 20:19:00.555584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.555570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location\") pod \"19629757-fe01-454f-8c69-43c69a7bad31\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") "
Sep 18 20:19:00.555737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.555633    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls\") pod \"19629757-fe01-454f-8c69-43c69a7bad31\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") "
Sep 18 20:19:00.555737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.555655    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config\") pod \"19629757-fe01-454f-8c69-43c69a7bad31\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") "
Sep 18 20:19:00.555737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.555673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert\") pod \"19629757-fe01-454f-8c69-43c69a7bad31\" (UID: \"19629757-fe01-454f-8c69-43c69a7bad31\") "
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556290    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19629757-fe01-454f-8c69-43c69a7bad31" (UID: "19629757-fe01-454f-8c69-43c69a7bad31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556309    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556424    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19629757-fe01-454f-8c69-43c69a7bad31" (UID: "19629757-fe01-454f-8c69-43c69a7bad31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556441    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556460    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19629757-fe01-454f-8c69-43c69a7bad31" (UID: "19629757-fe01-454f-8c69-43c69a7bad31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.556574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvg7\" (UniqueName: \"kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557378 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.557128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.557885 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.557849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.558594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558468    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19629757-fe01-454f-8c69-43c69a7bad31-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:00.558594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558474    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt" (OuterVolumeSpecName: "kube-api-access-s9cnt") pod "19629757-fe01-454f-8c69-43c69a7bad31" (UID: "19629757-fe01-454f-8c69-43c69a7bad31"). InnerVolumeSpecName "kube-api-access-s9cnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:00.558594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19629757-fe01-454f-8c69-43c69a7bad31" (UID: "19629757-fe01-454f-8c69-43c69a7bad31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:19:00.558594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558504    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:00.558594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558542    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19629757-fe01-454f-8c69-43c69a7bad31-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:00.563385 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.558946    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.563385 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.560738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.565635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.565612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvg7\" (UniqueName: \"kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.592577 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.592541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:00.659740 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.659716    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19629757-fe01-454f-8c69-43c69a7bad31-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:00.659740 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.659740    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9cnt\" (UniqueName: \"kubernetes.io/projected/19629757-fe01-454f-8c69-43c69a7bad31-kube-api-access-s9cnt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:00.801584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.801555    2573 generic.go:356] "Generic (PLEG): container finished" podID="19629757-fe01-454f-8c69-43c69a7bad31" containerID="49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb" exitCode=0
Sep 18 20:19:00.801755 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.801616    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"
Sep 18 20:19:00.801755 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.801629    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerDied","Data":"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"}
Sep 18 20:19:00.801755 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.801683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55" event={"ID":"19629757-fe01-454f-8c69-43c69a7bad31","Type":"ContainerDied","Data":"aa430bd18fd1a9e1994d90f5b093dc34e6f91a1acf6736fdd60dc96f2abb3dfb"}
Sep 18 20:19:00.801755 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.801720    2573 scope.go:117] "RemoveContainer" containerID="977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"
Sep 18 20:19:00.810058 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.810042    2573 scope.go:117] "RemoveContainer" containerID="49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"
Sep 18 20:19:00.817427 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.817406    2573 scope.go:117] "RemoveContainer" containerID="dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"
Sep 18 20:19:00.825209 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.825172    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:19:00.825541 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.825518    2573 scope.go:117] "RemoveContainer" containerID="977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"
Sep 18 20:19:00.825824 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:00.825806    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a\": container with ID starting with 977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a not found: ID does not exist" containerID="977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"
Sep 18 20:19:00.825878 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.825831    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a"} err="failed to get container status \"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a\": rpc error: code = NotFound desc = could not find container \"977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a\": container with ID starting with 977d33976a604921fb497cbb656ef24f627f269d00f125641a2d643242918a2a not found: ID does not exist"
Sep 18 20:19:00.825878 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.825845    2573 scope.go:117] "RemoveContainer" containerID="49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"
Sep 18 20:19:00.826261 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:00.826136    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb\": container with ID starting with 49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb not found: ID does not exist" containerID="49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"
Sep 18 20:19:00.826365 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.826261    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb"} err="failed to get container status \"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb\": rpc error: code = NotFound desc = could not find container \"49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb\": container with ID starting with 49bd5ed17e24202bc514b67d25ae0b5c4a3e4f14e0087269547c302e6c30a8cb not found: ID does not exist"
Sep 18 20:19:00.826365 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.826286    2573 scope.go:117] "RemoveContainer" containerID="dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"
Sep 18 20:19:00.826828 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:00.826681    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e\": container with ID starting with dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e not found: ID does not exist" containerID="dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"
Sep 18 20:19:00.826828 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.826715    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e"} err="failed to get container status \"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e\": rpc error: code = NotFound desc = could not find container \"dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e\": container with ID starting with dc78aef59ccf8e2626ce5ab6c9df292a92604948f270486d97b86ce099e04f0e not found: ID does not exist"
Sep 18 20:19:00.828368 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.828350    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w6r55"]
Sep 18 20:19:00.864447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:00.864429    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:00.866576 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:19:00.866553    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2b70f0_cf48_405f_818e_ed08b90300e0.slice/crio-9e056374852d9af0cbd2d609f587b27fb1010ca0f0b4aaf5252b5e352038834e WatchSource:0}: Error finding container 9e056374852d9af0cbd2d609f587b27fb1010ca0f0b4aaf5252b5e352038834e: Status 404 returned error can't find the container with id 9e056374852d9af0cbd2d609f587b27fb1010ca0f0b4aaf5252b5e352038834e
Sep 18 20:19:01.782893 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:01.782861    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19629757-fe01-454f-8c69-43c69a7bad31" path="/var/lib/kubelet/pods/19629757-fe01-454f-8c69-43c69a7bad31/volumes"
Sep 18 20:19:01.807696 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:01.807655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerStarted","Data":"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"}
Sep 18 20:19:01.807856 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:01.807719    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerStarted","Data":"9e056374852d9af0cbd2d609f587b27fb1010ca0f0b4aaf5252b5e352038834e"}
Sep 18 20:19:04.822521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:04.822486    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerID="7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504" exitCode=0
Sep 18 20:19:04.822880 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:04.822556    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerDied","Data":"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"}
Sep 18 20:19:05.829018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:05.828983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerStarted","Data":"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"}
Sep 18 20:19:05.829415 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:05.829042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerStarted","Data":"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"}
Sep 18 20:19:05.829415 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:05.829229    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:05.847768 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:05.847719    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" podStartSLOduration=2.090479993 podStartE2EDuration="5.8477003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757220307s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:19:05.846798672 +0000 UTC m=+2454.364768313" watchObservedRunningTime="2026-09-18 20:19:05.8477003 +0000 UTC m=+2454.365670001"
Sep 18 20:19:06.834046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:06.834020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:12.843679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:12.843653    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:42.847913 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:42.847877    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:50.340890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:50.340854    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:50.341471 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:50.341166    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kserve-container" containerID="cri-o://a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d" gracePeriod=30
Sep 18 20:19:50.341471 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:50.341209    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kube-rbac-proxy" containerID="cri-o://4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404" gracePeriod=30
Sep 18 20:19:51.007499 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.007469    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerID="4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404" exitCode=2
Sep 18 20:19:51.007685 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.007521    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerDied","Data":"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"}
Sep 18 20:19:51.272408 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.272384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:51.375419 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375389    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config\") pod \"fa2b70f0-cf48-405f-818e-ed08b90300e0\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") "
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375443    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert\") pod \"fa2b70f0-cf48-405f-818e-ed08b90300e0\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") "
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375490    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls\") pod \"fa2b70f0-cf48-405f-818e-ed08b90300e0\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") "
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375532    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location\") pod \"fa2b70f0-cf48-405f-818e-ed08b90300e0\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") "
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375565    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svvg7\" (UniqueName: \"kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7\") pod \"fa2b70f0-cf48-405f-818e-ed08b90300e0\" (UID: \"fa2b70f0-cf48-405f-818e-ed08b90300e0\") "
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375829    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa2b70f0-cf48-405f-818e-ed08b90300e0" (UID: "fa2b70f0-cf48-405f-818e-ed08b90300e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375860    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa2b70f0-cf48-405f-818e-ed08b90300e0" (UID: "fa2b70f0-cf48-405f-818e-ed08b90300e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:19:51.375898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.375885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa2b70f0-cf48-405f-818e-ed08b90300e0" (UID: "fa2b70f0-cf48-405f-818e-ed08b90300e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:19:51.377690 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.377667    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7" (OuterVolumeSpecName: "kube-api-access-svvg7") pod "fa2b70f0-cf48-405f-818e-ed08b90300e0" (UID: "fa2b70f0-cf48-405f-818e-ed08b90300e0"). InnerVolumeSpecName "kube-api-access-svvg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:19:51.377756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.377740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa2b70f0-cf48-405f-818e-ed08b90300e0" (UID: "fa2b70f0-cf48-405f-818e-ed08b90300e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:19:51.476302 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.476274    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa2b70f0-cf48-405f-818e-ed08b90300e0-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:51.476302 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.476296    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa2b70f0-cf48-405f-818e-ed08b90300e0-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:51.476302 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.476306    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvg7\" (UniqueName: \"kubernetes.io/projected/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-api-access-svvg7\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:51.476493 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.476317    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:51.476493 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:51.476326    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa2b70f0-cf48-405f-818e-ed08b90300e0-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:19:52.013135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.013105    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerID="a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d" exitCode=0
Sep 18 20:19:52.013327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.013142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerDied","Data":"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"}
Sep 18 20:19:52.013327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.013169    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"
Sep 18 20:19:52.013327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.013187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t" event={"ID":"fa2b70f0-cf48-405f-818e-ed08b90300e0","Type":"ContainerDied","Data":"9e056374852d9af0cbd2d609f587b27fb1010ca0f0b4aaf5252b5e352038834e"}
Sep 18 20:19:52.013327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.013236    2573 scope.go:117] "RemoveContainer" containerID="4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"
Sep 18 20:19:52.021256 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.021237    2573 scope.go:117] "RemoveContainer" containerID="a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"
Sep 18 20:19:52.028341 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.028327    2573 scope.go:117] "RemoveContainer" containerID="7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"
Sep 18 20:19:52.031138 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.031116    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:52.035926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.035904    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pf49t"]
Sep 18 20:19:52.036598 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.036581    2573 scope.go:117] "RemoveContainer" containerID="4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"
Sep 18 20:19:52.036859 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:52.036838    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404\": container with ID starting with 4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404 not found: ID does not exist" containerID="4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"
Sep 18 20:19:52.036922 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.036866    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404"} err="failed to get container status \"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404\": rpc error: code = NotFound desc = could not find container \"4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404\": container with ID starting with 4bd880a0a0f193bfb0f3f51ee9e82ead3146f33851f16f722a2f26c0e7c46404 not found: ID does not exist"
Sep 18 20:19:52.036922 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.036882    2573 scope.go:117] "RemoveContainer" containerID="a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"
Sep 18 20:19:52.037120 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:52.037106    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d\": container with ID starting with a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d not found: ID does not exist" containerID="a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"
Sep 18 20:19:52.037161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.037122    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d"} err="failed to get container status \"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d\": rpc error: code = NotFound desc = could not find container \"a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d\": container with ID starting with a771b66226258cc8dbf58c3ede20b9a8dad45c74a60ca51e5e9765861be1ee8d not found: ID does not exist"
Sep 18 20:19:52.037161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.037135    2573 scope.go:117] "RemoveContainer" containerID="7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"
Sep 18 20:19:52.037393 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:19:52.037376    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504\": container with ID starting with 7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504 not found: ID does not exist" containerID="7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"
Sep 18 20:19:52.037445 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:52.037401    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504"} err="failed to get container status \"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504\": rpc error: code = NotFound desc = could not find container \"7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504\": container with ID starting with 7af610e575224c0df2aa8ab89967444df1ca335f79e0c1c88aea9f6e56529504 not found: ID does not exist"
Sep 18 20:19:53.782348 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:53.782319    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" path="/var/lib/kubelet/pods/fa2b70f0-cf48-405f-818e-ed08b90300e0/volumes"
Sep 18 20:19:55.519410 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519622    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="storage-initializer"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519633    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="storage-initializer"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519650    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kube-rbac-proxy"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519655    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kube-rbac-proxy"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519664    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519677    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="storage-initializer"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519682    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="storage-initializer"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519688    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519702    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519711    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kube-rbac-proxy"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519716    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kube-rbac-proxy"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519759    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kube-rbac-proxy"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519767    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519773    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2b70f0-cf48-405f-818e-ed08b90300e0" containerName="kserve-container"
Sep 18 20:19:55.519771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.519780    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="19629757-fe01-454f-8c69-43c69a7bad31" containerName="kube-rbac-proxy"
Sep 18 20:19:55.524408 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.524390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.526738 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.526706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:19:55.526852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.526761    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:19:55.526852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.526782    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:19:55.526852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.526764    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 20:19:55.527459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.527421    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:19:55.527459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.527437    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 18 20:19:55.527647 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.527476    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:19:55.532546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.532523    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:19:55.711792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.711761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.711792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.711793    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.712008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.711824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.712008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.711943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.712109 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.712000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7zf\" (UniqueName: \"kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.812962 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.812902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.812962 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.812943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.812971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7zf\" (UniqueName: \"kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.813004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.813020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.813425    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813659 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.813629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.813766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.813679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.815618 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.815601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.820587 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.820565    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7zf\" (UniqueName: \"kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:55.835323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:55.835305    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:19:56.105897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:56.105867    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:19:56.108482 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:19:56.108454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a9d22e_cff1_414e_847e_d754bc7defc7.slice/crio-f6d14e82a104619dc73ec6100389d05b8e7c138dc3bf7d8114b214dc3f7201c9 WatchSource:0}: Error finding container f6d14e82a104619dc73ec6100389d05b8e7c138dc3bf7d8114b214dc3f7201c9: Status 404 returned error can't find the container with id f6d14e82a104619dc73ec6100389d05b8e7c138dc3bf7d8114b214dc3f7201c9
Sep 18 20:19:57.034059 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:57.034022    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerStarted","Data":"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"}
Sep 18 20:19:57.034449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:19:57.034078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerStarted","Data":"f6d14e82a104619dc73ec6100389d05b8e7c138dc3bf7d8114b214dc3f7201c9"}
Sep 18 20:20:00.047271 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:00.047161    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerID="1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa" exitCode=0
Sep 18 20:20:00.047271 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:00.047234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerDied","Data":"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"}
Sep 18 20:20:01.054718 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:01.054684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerStarted","Data":"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"}
Sep 18 20:20:01.055095 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:01.054781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerStarted","Data":"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"}
Sep 18 20:20:01.055095 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:01.054979    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:01.073450 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:01.073390    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podStartSLOduration=2.452861149 podStartE2EDuration="6.073371356s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.620510207s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:01.072231264 +0000 UTC m=+2509.590200907" watchObservedRunningTime="2026-09-18 20:20:01.073371356 +0000 UTC m=+2509.591341000"
Sep 18 20:20:02.059255 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:02.059222    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:02.060648 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:02.060616    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:20:03.063746 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:03.063712    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:20:08.068134 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:08.068104    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:08.068627 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:08.068601    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:20:18.069312 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:18.069283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:25.623887 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:25.623855    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:20:25.624449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:25.624119    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container" containerID="cri-o://c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86" gracePeriod=30
Sep 18 20:20:25.624449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:25.624177    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kube-rbac-proxy" containerID="cri-o://10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc" gracePeriod=30
Sep 18 20:20:26.167630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.167593    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerID="10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc" exitCode=2
Sep 18 20:20:26.167804 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.167646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerDied","Data":"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"}
Sep 18 20:20:26.364069 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.364045    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:26.465174 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465102    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh7zf\" (UniqueName: \"kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf\") pod \"a8a9d22e-cff1-414e-847e-d754bc7defc7\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") "
Sep 18 20:20:26.465174 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config\") pod \"a8a9d22e-cff1-414e-847e-d754bc7defc7\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") "
Sep 18 20:20:26.465380 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465238    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls\") pod \"a8a9d22e-cff1-414e-847e-d754bc7defc7\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") "
Sep 18 20:20:26.465380 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465283    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location\") pod \"a8a9d22e-cff1-414e-847e-d754bc7defc7\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") "
Sep 18 20:20:26.465380 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465309    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert\") pod \"a8a9d22e-cff1-414e-847e-d754bc7defc7\" (UID: \"a8a9d22e-cff1-414e-847e-d754bc7defc7\") "
Sep 18 20:20:26.465630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465599    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8a9d22e-cff1-414e-847e-d754bc7defc7" (UID: "a8a9d22e-cff1-414e-847e-d754bc7defc7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:20:26.465736 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465554    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8a9d22e-cff1-414e-847e-d754bc7defc7" (UID: "a8a9d22e-cff1-414e-847e-d754bc7defc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:20:26.465736 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.465693    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8a9d22e-cff1-414e-847e-d754bc7defc7" (UID: "a8a9d22e-cff1-414e-847e-d754bc7defc7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:20:26.467181 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.467160    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf" (OuterVolumeSpecName: "kube-api-access-dh7zf") pod "a8a9d22e-cff1-414e-847e-d754bc7defc7" (UID: "a8a9d22e-cff1-414e-847e-d754bc7defc7"). InnerVolumeSpecName "kube-api-access-dh7zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:20:26.467287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.467236    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8a9d22e-cff1-414e-847e-d754bc7defc7" (UID: "a8a9d22e-cff1-414e-847e-d754bc7defc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:20:26.566739 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.566708    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a9d22e-cff1-414e-847e-d754bc7defc7-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:20:26.566739 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.566734    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:20:26.566906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.566745    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7zf\" (UniqueName: \"kubernetes.io/projected/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-api-access-dh7zf\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:20:26.566906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.566755    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a9d22e-cff1-414e-847e-d754bc7defc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:20:26.566906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:26.566765    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a9d22e-cff1-414e-847e-d754bc7defc7-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:20:27.173784 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.173750    2573 generic.go:356] "Generic (PLEG): container finished" podID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerID="c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86" exitCode=0
Sep 18 20:20:27.174195 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.173804    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"
Sep 18 20:20:27.174195 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.173830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerDied","Data":"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"}
Sep 18 20:20:27.174195 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.173890    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4" event={"ID":"a8a9d22e-cff1-414e-847e-d754bc7defc7","Type":"ContainerDied","Data":"f6d14e82a104619dc73ec6100389d05b8e7c138dc3bf7d8114b214dc3f7201c9"}
Sep 18 20:20:27.174195 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.173916    2573 scope.go:117] "RemoveContainer" containerID="10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"
Sep 18 20:20:27.182311 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.182295    2573 scope.go:117] "RemoveContainer" containerID="c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"
Sep 18 20:20:27.189380 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.189364    2573 scope.go:117] "RemoveContainer" containerID="1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"
Sep 18 20:20:27.194265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.194242    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:20:27.197144 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197127    2573 scope.go:117] "RemoveContainer" containerID="10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"
Sep 18 20:20:27.197443 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:20:27.197421    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc\": container with ID starting with 10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc not found: ID does not exist" containerID="10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"
Sep 18 20:20:27.197515 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197450    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc"} err="failed to get container status \"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc\": rpc error: code = NotFound desc = could not find container \"10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc\": container with ID starting with 10c69ad64ff3830eb35a5fb704bdac075f509d40e12e9a51ded6d1d44b8354dc not found: ID does not exist"
Sep 18 20:20:27.197515 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197471    2573 scope.go:117] "RemoveContainer" containerID="c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"
Sep 18 20:20:27.197635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197579    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-xffd4"]
Sep 18 20:20:27.197744 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:20:27.197726    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86\": container with ID starting with c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86 not found: ID does not exist" containerID="c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"
Sep 18 20:20:27.197785 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197749    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86"} err="failed to get container status \"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86\": rpc error: code = NotFound desc = could not find container \"c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86\": container with ID starting with c5fb6a7b9d3317c14c1a7e8d4b21eddadd6a32eca5358b5f1992e733d7a42f86 not found: ID does not exist"
Sep 18 20:20:27.197785 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.197764    2573 scope.go:117] "RemoveContainer" containerID="1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"
Sep 18 20:20:27.198003 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:20:27.197988    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa\": container with ID starting with 1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa not found: ID does not exist" containerID="1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"
Sep 18 20:20:27.198044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.198006    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa"} err="failed to get container status \"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa\": rpc error: code = NotFound desc = could not find container \"1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa\": container with ID starting with 1a2e3c19004ed4858719e82fff976e4f80f8edd6098b88488fb84e05b70c3ffa not found: ID does not exist"
Sep 18 20:20:27.781934 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:27.781905    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" path="/var/lib/kubelet/pods/a8a9d22e-cff1-414e-847e-d754bc7defc7/volumes"
Sep 18 20:20:30.808664 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808631    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808931    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808942    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808950    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kube-rbac-proxy"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808955    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kube-rbac-proxy"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808962    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="storage-initializer"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.808967    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="storage-initializer"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.809006    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kserve-container"
Sep 18 20:20:30.809025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.809015    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a9d22e-cff1-414e-847e-d754bc7defc7" containerName="kube-rbac-proxy"
Sep 18 20:20:30.819541 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.819510    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:30.821711 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.821683    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 18 20:20:30.821908 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.821895    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:20:30.822133 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822117    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:20:30.822252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822169    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:20:30.822318 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822289    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:20:30.822360 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:20:30.822567 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822549    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:20:30.822678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.822594    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:20:30.900111 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.900086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:30.900262 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.900123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:30.900262 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.900166    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sxc\" (UniqueName: \"kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:30.900262 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.900225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:30.900381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:30.900275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001473 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001601 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sxc\" (UniqueName: \"kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.001994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.001974    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.002140 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.002122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.002244 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.002222    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.004131 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.004107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.009404 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.009381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sxc\" (UniqueName: \"kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.131133 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.131104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:31.399361 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:31.399332    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:20:31.401253 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:20:31.401224    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce456de_1514_4b2e_a572_64e03e12ab09.slice/crio-1cbd6e68efe8689cfd53387529c3aa6040e0d7415f391fd05f1e4e878b929a97 WatchSource:0}: Error finding container 1cbd6e68efe8689cfd53387529c3aa6040e0d7415f391fd05f1e4e878b929a97: Status 404 returned error can't find the container with id 1cbd6e68efe8689cfd53387529c3aa6040e0d7415f391fd05f1e4e878b929a97
Sep 18 20:20:32.197802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:32.197768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerStarted","Data":"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"}
Sep 18 20:20:32.198184 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:32.197832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerStarted","Data":"1cbd6e68efe8689cfd53387529c3aa6040e0d7415f391fd05f1e4e878b929a97"}
Sep 18 20:20:36.215785 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:36.215751    2573 generic.go:356] "Generic (PLEG): container finished" podID="cce456de-1514-4b2e-a572-64e03e12ab09" containerID="e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757" exitCode=0
Sep 18 20:20:36.216134 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:36.215827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerDied","Data":"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"}
Sep 18 20:20:37.222424 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:37.222384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerStarted","Data":"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"}
Sep 18 20:20:37.222869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:37.222451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerStarted","Data":"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"}
Sep 18 20:20:37.222869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:37.222775    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:37.222869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:37.222844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:37.242477 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:37.242428    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" podStartSLOduration=3.270289279 podStartE2EDuration="7.242412628s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972123349s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:37.240133745 +0000 UTC m=+2545.758103385" watchObservedRunningTime="2026-09-18 20:20:37.242412628 +0000 UTC m=+2545.760382270"
Sep 18 20:20:43.231813 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:43.231786    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:20:46.254676 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.254646    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:20:46.258055 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.258034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.260360 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.260337    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 18 20:20:46.260360 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.260362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:20:46.260556 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.260408    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 18 20:20:46.266847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.266827    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:20:46.434807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.434770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.434998 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.434822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.434998 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.434892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.434998 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.434945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z627t\" (UniqueName: \"kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.535947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z627t\" (UniqueName: \"kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.535993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.536015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.536037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536429 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.536408    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.536703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.536674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.538638 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.538616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.544267 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.544246    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z627t\" (UniqueName: \"kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t\") pod \"isvc-sklearn-s3-predictor-6c7db45b94-xfs6b\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.569679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.569653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:46.837680 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:46.837623    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:20:46.839939 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:20:46.839909    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7b5bb4_25a0_4496_803f_ed2322c4e84a.slice/crio-6284bc4d92c21464f669884467f5cda1a2d0c359d0bcf3aa203380ddb7648cbb WatchSource:0}: Error finding container 6284bc4d92c21464f669884467f5cda1a2d0c359d0bcf3aa203380ddb7648cbb: Status 404 returned error can't find the container with id 6284bc4d92c21464f669884467f5cda1a2d0c359d0bcf3aa203380ddb7648cbb
Sep 18 20:20:47.268677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:47.268640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerStarted","Data":"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"}
Sep 18 20:20:47.269053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:47.268702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerStarted","Data":"6284bc4d92c21464f669884467f5cda1a2d0c359d0bcf3aa203380ddb7648cbb"}
Sep 18 20:20:48.275237 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:48.275188    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerID="e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a" exitCode=0
Sep 18 20:20:48.275237 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:48.275229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerDied","Data":"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"}
Sep 18 20:20:49.281384 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:49.281349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerStarted","Data":"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"}
Sep 18 20:20:49.281765 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:49.281409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerStarted","Data":"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"}
Sep 18 20:20:49.281765 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:49.281435    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:49.318796 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:49.318749    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podStartSLOduration=2.078624193 podStartE2EDuration="3.318731573s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.24010738s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:20:49.317985597 +0000 UTC m=+2557.835955238" watchObservedRunningTime="2026-09-18 20:20:49.318731573 +0000 UTC m=+2557.836701213"
Sep 18 20:20:50.285235 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:50.285188    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:50.286583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:50.286558    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:20:51.289607 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:51.289574    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:20:56.301216 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:56.301175    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:20:56.301630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:20:56.301603    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:06.302241 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:06.302183    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:13.236023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:13.235988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:21:16.301873 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:16.301831    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:20.863906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:20.863871    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:21:20.864280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:20.864153    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kserve-container" containerID="cri-o://5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f" gracePeriod=30
Sep 18 20:21:20.864280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:20.864248    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kube-rbac-proxy" containerID="cri-o://a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e" gracePeriod=30
Sep 18 20:21:21.415297 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:21.415262    2573 generic.go:356] "Generic (PLEG): container finished" podID="cce456de-1514-4b2e-a572-64e03e12ab09" containerID="a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e" exitCode=2
Sep 18 20:21:21.415465 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:21.415336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerDied","Data":"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"}
Sep 18 20:21:22.003724 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.003700    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:21:22.112031 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls\") pod \"cce456de-1514-4b2e-a572-64e03e12ab09\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") "
Sep 18 20:21:22.112225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112085    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7sxc\" (UniqueName: \"kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc\") pod \"cce456de-1514-4b2e-a572-64e03e12ab09\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") "
Sep 18 20:21:22.112225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112114    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config\") pod \"cce456de-1514-4b2e-a572-64e03e12ab09\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") "
Sep 18 20:21:22.112225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112132    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location\") pod \"cce456de-1514-4b2e-a572-64e03e12ab09\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") "
Sep 18 20:21:22.112225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112154    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert\") pod \"cce456de-1514-4b2e-a572-64e03e12ab09\" (UID: \"cce456de-1514-4b2e-a572-64e03e12ab09\") "
Sep 18 20:21:22.112552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cce456de-1514-4b2e-a572-64e03e12ab09" (UID: "cce456de-1514-4b2e-a572-64e03e12ab09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:22.112613 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112583    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cce456de-1514-4b2e-a572-64e03e12ab09" (UID: "cce456de-1514-4b2e-a572-64e03e12ab09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:22.112675 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.112588    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cce456de-1514-4b2e-a572-64e03e12ab09" (UID: "cce456de-1514-4b2e-a572-64e03e12ab09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:21:22.114215 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.114176    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cce456de-1514-4b2e-a572-64e03e12ab09" (UID: "cce456de-1514-4b2e-a572-64e03e12ab09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:22.114316 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.114226    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc" (OuterVolumeSpecName: "kube-api-access-l7sxc") pod "cce456de-1514-4b2e-a572-64e03e12ab09" (UID: "cce456de-1514-4b2e-a572-64e03e12ab09"). InnerVolumeSpecName "kube-api-access-l7sxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:22.213471 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.213450    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7sxc\" (UniqueName: \"kubernetes.io/projected/cce456de-1514-4b2e-a572-64e03e12ab09-kube-api-access-l7sxc\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:21:22.213471 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.213471    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:21:22.213611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.213480    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce456de-1514-4b2e-a572-64e03e12ab09-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:21:22.213611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.213489    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce456de-1514-4b2e-a572-64e03e12ab09-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:21:22.213611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.213498    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce456de-1514-4b2e-a572-64e03e12ab09-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:21:22.421693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.421622    2573 generic.go:356] "Generic (PLEG): container finished" podID="cce456de-1514-4b2e-a572-64e03e12ab09" containerID="5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f" exitCode=0
Sep 18 20:21:22.421693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.421683    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"
Sep 18 20:21:22.421847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.421700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerDied","Data":"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"}
Sep 18 20:21:22.421847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.421774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf" event={"ID":"cce456de-1514-4b2e-a572-64e03e12ab09","Type":"ContainerDied","Data":"1cbd6e68efe8689cfd53387529c3aa6040e0d7415f391fd05f1e4e878b929a97"}
Sep 18 20:21:22.421847 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.421813    2573 scope.go:117] "RemoveContainer" containerID="a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"
Sep 18 20:21:22.431048 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.431032    2573 scope.go:117] "RemoveContainer" containerID="5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"
Sep 18 20:21:22.438545 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.438527    2573 scope.go:117] "RemoveContainer" containerID="e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"
Sep 18 20:21:22.442523 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.442501    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:21:22.445947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.445858    2573 scope.go:117] "RemoveContainer" containerID="a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"
Sep 18 20:21:22.446589 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:21:22.446263    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e\": container with ID starting with a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e not found: ID does not exist" containerID="a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"
Sep 18 20:21:22.446589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.446294    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e"} err="failed to get container status \"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e\": rpc error: code = NotFound desc = could not find container \"a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e\": container with ID starting with a6c7d473fc40675e1ee2821f4ee58e4d4ee33fe49b4f94277dfa05be7aa6d88e not found: ID does not exist"
Sep 18 20:21:22.446589 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.446317    2573 scope.go:117] "RemoveContainer" containerID="5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"
Sep 18 20:21:22.446734 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:21:22.446720    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f\": container with ID starting with 5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f not found: ID does not exist" containerID="5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"
Sep 18 20:21:22.446775 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.446741    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f"} err="failed to get container status \"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f\": rpc error: code = NotFound desc = could not find container \"5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f\": container with ID starting with 5274eb81979c888227df1b5cd81d405668435d407091c196e0930f378a0b912f not found: ID does not exist"
Sep 18 20:21:22.446775 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.446756    2573 scope.go:117] "RemoveContainer" containerID="e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"
Sep 18 20:21:22.447025 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:21:22.447008    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757\": container with ID starting with e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757 not found: ID does not exist" containerID="e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"
Sep 18 20:21:22.447080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.447030    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757"} err="failed to get container status \"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757\": rpc error: code = NotFound desc = could not find container \"e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757\": container with ID starting with e20d7933409e513b151478d8fe2bcacf918c71278fb068d4a4ca370bf7f30757 not found: ID does not exist"
Sep 18 20:21:22.448044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:22.448024    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-psxwf"]
Sep 18 20:21:23.782092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:23.782060    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" path="/var/lib/kubelet/pods/cce456de-1514-4b2e-a572-64e03e12ab09/volumes"
Sep 18 20:21:26.054501 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054428    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054728    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kube-rbac-proxy"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054738    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kube-rbac-proxy"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054747    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kserve-container"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054753    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kserve-container"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054764    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="storage-initializer"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054769    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="storage-initializer"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054829    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kserve-container"
Sep 18 20:21:26.054837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.054837    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce456de-1514-4b2e-a572-64e03e12ab09" containerName="kube-rbac-proxy"
Sep 18 20:21:26.059617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.059599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.061705 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.061686    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 18 20:21:26.061817 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.061704    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 18 20:21:26.061817 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.061691    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:21:26.061817 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.061708    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:21:26.068165 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.068144    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:21:26.142251 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.142223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9g4\" (UniqueName: \"kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.142251 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.142256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.142448 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.142273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.142448 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.142337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.142448 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.142354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243411 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9g4\" (UniqueName: \"kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243414    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243434    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.243746 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:21:26.243596    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 18 20:21:26.243746 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:21:26.243692    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls podName:07940e22-4a6d-4de4-93db-f279977c3388 nodeName:}" failed. No retries permitted until 2026-09-18 20:21:26.74366996 +0000 UTC m=+2595.261639592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls") pod "isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" (UID: "07940e22-4a6d-4de4-93db-f279977c3388") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 18 20:21:26.243856 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.243830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.244266 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.244245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.244322 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.244245    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.253560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.253541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9g4\" (UniqueName: \"kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.302362 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.302335    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:26.747486 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.747449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.749825 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.749798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:26.970537 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:26.970503    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:27.242919 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:27.242895    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:21:27.244932 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:21:27.244906    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07940e22_4a6d_4de4_93db_f279977c3388.slice/crio-3c0d4cb7c48e72fe8f75f9bcea183649b44c2ce8ab9b0e5570790f736312efe8 WatchSource:0}: Error finding container 3c0d4cb7c48e72fe8f75f9bcea183649b44c2ce8ab9b0e5570790f736312efe8: Status 404 returned error can't find the container with id 3c0d4cb7c48e72fe8f75f9bcea183649b44c2ce8ab9b0e5570790f736312efe8
Sep 18 20:21:27.444288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:27.444247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerStarted","Data":"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"}
Sep 18 20:21:27.444459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:27.444326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerStarted","Data":"3c0d4cb7c48e72fe8f75f9bcea183649b44c2ce8ab9b0e5570790f736312efe8"}
Sep 18 20:21:31.461277 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:31.461237    2573 generic.go:356] "Generic (PLEG): container finished" podID="07940e22-4a6d-4de4-93db-f279977c3388" containerID="12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239" exitCode=0
Sep 18 20:21:31.461655 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:31.461295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerDied","Data":"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"}
Sep 18 20:21:32.469331 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:32.469232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerStarted","Data":"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"}
Sep 18 20:21:34.479995 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:34.479959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerStarted","Data":"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"}
Sep 18 20:21:34.480403 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:34.480023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerStarted","Data":"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"}
Sep 18 20:21:34.480403 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:34.480106    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:34.480403 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:34.480168    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:34.500711 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:34.500663    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podStartSLOduration=2.043446692 podStartE2EDuration="8.500646748s" totalImagesPullingTime="2.760999662s" totalInitContainerRuntime="3.696200394s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:21:34.498936133 +0000 UTC m=+2603.016905773" watchObservedRunningTime="2026-09-18 20:21:34.500646748 +0000 UTC m=+2603.018616387"
Sep 18 20:21:35.484476 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:35.484445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:36.302069 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:36.302025    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:41.493708 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:41.493677    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:21:46.301977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:46.301937    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:21:56.301997 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:21:56.301949    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:22:01.495511 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:01.495465    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:22:06.302358 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:06.302326    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:22:11.496284 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:11.496253    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:22:16.326295 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.326249    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:22:16.326772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.326538    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container" containerID="cri-o://754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da" gracePeriod=30
Sep 18 20:22:16.326772 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.326557    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kube-rbac-proxy" containerID="cri-o://9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772" gracePeriod=30
Sep 18 20:22:16.514822 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.514789    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:22:16.517343 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.517323    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.519987 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.519968    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:22:16.520095 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.519967    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 18 20:22:16.527602 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.527579    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:22:16.553850 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.553826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.553949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.553856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.553949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.553878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rl5k\" (UniqueName: \"kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.554029 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.553975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.554029 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.554003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.652807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.652781    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerID="9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772" exitCode=2
Sep 18 20:22:16.652907 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.652854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerDied","Data":"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"}
Sep 18 20:22:16.655116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655240 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655240 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655240 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655240 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rl5k\" (UniqueName: \"kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655670 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.655978 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.656019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.655959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.657494 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.657479    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.663551 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.663525    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rl5k\" (UniqueName: \"kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:16.828061 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:16.828029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:17.099444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:17.099421    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:22:17.101628 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:22:17.101605    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e0b5fb_5064_42db_8fc6_f58edee5d764.slice/crio-234e79deedefaa130cea86923bc0960d49944a0877c02366dce7cd938bcfb777 WatchSource:0}: Error finding container 234e79deedefaa130cea86923bc0960d49944a0877c02366dce7cd938bcfb777: Status 404 returned error can't find the container with id 234e79deedefaa130cea86923bc0960d49944a0877c02366dce7cd938bcfb777
Sep 18 20:22:17.658467 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:17.658431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerStarted","Data":"abc8c8154c9b020d350bb16547865f2b2d1846d5f2d4cb6fcf4801caa397af6d"}
Sep 18 20:22:17.658840 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:17.658515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerStarted","Data":"234e79deedefaa130cea86923bc0960d49944a0877c02366dce7cd938bcfb777"}
Sep 18 20:22:18.664892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:18.664854    2573 generic.go:356] "Generic (PLEG): container finished" podID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerID="abc8c8154c9b020d350bb16547865f2b2d1846d5f2d4cb6fcf4801caa397af6d" exitCode=0
Sep 18 20:22:18.665373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:18.664912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerDied","Data":"abc8c8154c9b020d350bb16547865f2b2d1846d5f2d4cb6fcf4801caa397af6d"}
Sep 18 20:22:19.671390 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.671357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerStarted","Data":"9f69411000f967241de480332fec80527c329c3f001c858eb329547b74756f89"}
Sep 18 20:22:19.671756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.671412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerStarted","Data":"4fbd1f949ca5b82d71a2f7f869fb6f3723eb85b2fd25afc284dd4df18d544dbb"}
Sep 18 20:22:19.671756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.671561    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:19.671756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.671670    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:19.672957 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.672931    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:19.691625 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:19.691583    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podStartSLOduration=2.6684001999999998 podStartE2EDuration="3.691569598s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.023169398s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:22:19.689080317 +0000 UTC m=+2648.207049957" watchObservedRunningTime="2026-09-18 20:22:19.691569598 +0000 UTC m=+2648.209539237"
Sep 18 20:22:20.675976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:20.675938    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:21.062857 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.062833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:22:21.094413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094388    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config\") pod \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") "
Sep 18 20:22:21.094550 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094439    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location\") pod \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") "
Sep 18 20:22:21.094550 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z627t\" (UniqueName: \"kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t\") pod \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") "
Sep 18 20:22:21.094550 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094511    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls\") pod \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\" (UID: \"aa7b5bb4-25a0-4496-803f-ed2322c4e84a\") "
Sep 18 20:22:21.094746 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094727    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa7b5bb4-25a0-4496-803f-ed2322c4e84a" (UID: "aa7b5bb4-25a0-4496-803f-ed2322c4e84a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:22:21.094814 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.094792    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa7b5bb4-25a0-4496-803f-ed2322c4e84a" (UID: "aa7b5bb4-25a0-4496-803f-ed2322c4e84a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:22:21.096563 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.096534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t" (OuterVolumeSpecName: "kube-api-access-z627t") pod "aa7b5bb4-25a0-4496-803f-ed2322c4e84a" (UID: "aa7b5bb4-25a0-4496-803f-ed2322c4e84a"). InnerVolumeSpecName "kube-api-access-z627t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:22:21.096655 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.096557    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa7b5bb4-25a0-4496-803f-ed2322c4e84a" (UID: "aa7b5bb4-25a0-4496-803f-ed2322c4e84a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:22:21.195778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.195717    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:22:21.195778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.195738    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z627t\" (UniqueName: \"kubernetes.io/projected/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-api-access-z627t\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:22:21.195778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.195748    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:22:21.195778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.195756    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa7b5bb4-25a0-4496-803f-ed2322c4e84a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:22:21.682051 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.682018    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerID="754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da" exitCode=0
Sep 18 20:22:21.682491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.682058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerDied","Data":"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"}
Sep 18 20:22:21.682491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.682067    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"
Sep 18 20:22:21.682491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.682104    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b" event={"ID":"aa7b5bb4-25a0-4496-803f-ed2322c4e84a","Type":"ContainerDied","Data":"6284bc4d92c21464f669884467f5cda1a2d0c359d0bcf3aa203380ddb7648cbb"}
Sep 18 20:22:21.682491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.682131    2573 scope.go:117] "RemoveContainer" containerID="9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"
Sep 18 20:22:21.690568 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.690550    2573 scope.go:117] "RemoveContainer" containerID="754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"
Sep 18 20:22:21.697964 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.697942    2573 scope.go:117] "RemoveContainer" containerID="e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"
Sep 18 20:22:21.702676 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.702652    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:22:21.705649 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.705635    2573 scope.go:117] "RemoveContainer" containerID="9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"
Sep 18 20:22:21.705937 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:22:21.705920    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772\": container with ID starting with 9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772 not found: ID does not exist" containerID="9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"
Sep 18 20:22:21.705990 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.705947    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772"} err="failed to get container status \"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772\": rpc error: code = NotFound desc = could not find container \"9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772\": container with ID starting with 9ea6c3fe45309a07ab64bd024a44928646aac344f936e8e50a50ffc3fa4f1772 not found: ID does not exist"
Sep 18 20:22:21.705990 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.705965    2573 scope.go:117] "RemoveContainer" containerID="754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"
Sep 18 20:22:21.706174 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.706156    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c7db45b94-xfs6b"]
Sep 18 20:22:21.706247 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:22:21.706232    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da\": container with ID starting with 754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da not found: ID does not exist" containerID="754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"
Sep 18 20:22:21.706287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.706251    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da"} err="failed to get container status \"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da\": rpc error: code = NotFound desc = could not find container \"754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da\": container with ID starting with 754353ce5e3672b0c22652a42c9c99f60e938df424de65c923169599d91460da not found: ID does not exist"
Sep 18 20:22:21.706287 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.706265    2573 scope.go:117] "RemoveContainer" containerID="e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"
Sep 18 20:22:21.706520 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:22:21.706502    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a\": container with ID starting with e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a not found: ID does not exist" containerID="e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"
Sep 18 20:22:21.706562 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.706528    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a"} err="failed to get container status \"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a\": rpc error: code = NotFound desc = could not find container \"e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a\": container with ID starting with e1ee122114e29281a0b0013392d3034b6c0d2044433324982a62294dd148c96a not found: ID does not exist"
Sep 18 20:22:21.786786 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:21.786751    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" path="/var/lib/kubelet/pods/aa7b5bb4-25a0-4496-803f-ed2322c4e84a/volumes"
Sep 18 20:22:25.680300 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:25.680270    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:22:25.680831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:25.680807    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:35.681253 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:35.681196    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:41.497684 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:41.497649    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:22:45.680759 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:45.680724    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:46.083512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.083436    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:22:46.083766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.083717    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container" containerID="cri-o://6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476" gracePeriod=30
Sep 18 20:22:46.083832 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.083773    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" containerID="cri-o://9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef" gracePeriod=30
Sep 18 20:22:46.083832 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.083814    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-agent" containerID="cri-o://1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375" gracePeriod=30
Sep 18 20:22:46.488406 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.488365    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:22:46.793194 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.793106    2573 generic.go:356] "Generic (PLEG): container finished" podID="07940e22-4a6d-4de4-93db-f279977c3388" containerID="9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef" exitCode=2
Sep 18 20:22:46.793194 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:46.793157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerDied","Data":"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"}
Sep 18 20:22:48.805299 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:48.805265    2573 generic.go:356] "Generic (PLEG): container finished" podID="07940e22-4a6d-4de4-93db-f279977c3388" containerID="6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476" exitCode=0
Sep 18 20:22:48.805657 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:48.805334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerDied","Data":"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"}
Sep 18 20:22:51.254035 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.253999    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:22:51.254427 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254395    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container"
Sep 18 20:22:51.254427 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254410    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container"
Sep 18 20:22:51.254508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254429    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="storage-initializer"
Sep 18 20:22:51.254508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254436    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="storage-initializer"
Sep 18 20:22:51.254508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254449    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kube-rbac-proxy"
Sep 18 20:22:51.254508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254454    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kube-rbac-proxy"
Sep 18 20:22:51.254508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254506    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kserve-container"
Sep 18 20:22:51.254660 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.254513    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7b5bb4-25a0-4496-803f-ed2322c4e84a" containerName="kube-rbac-proxy"
Sep 18 20:22:51.256764 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.256749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.258787 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.258768    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 18 20:22:51.259050 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.259031    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 18 20:22:51.267984 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.267964    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:22:51.347161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.346973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.347161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.347047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.347161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.347122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.347460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.347260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.347460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.347376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tnb\" (UniqueName: \"kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448093 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tnb\" (UniqueName: \"kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448123    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448643 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448620    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448784 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.448837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.448801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.450674 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.450652    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.457258 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.457235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tnb\" (UniqueName: \"kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5c6v8\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.488399 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.488368    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:22:51.494413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.494391    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:22:51.567305 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.567221    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:22:51.837063 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:51.837040    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:22:51.838974 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:22:51.838932    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fe9711_2034_4f90_b6c0_fee97a58f2e6.slice/crio-355658202d7cc345183093f26d34ad6d8ddc6b00ae7d9d149cbdf977a40f0065 WatchSource:0}: Error finding container 355658202d7cc345183093f26d34ad6d8ddc6b00ae7d9d149cbdf977a40f0065: Status 404 returned error can't find the container with id 355658202d7cc345183093f26d34ad6d8ddc6b00ae7d9d149cbdf977a40f0065
Sep 18 20:22:52.823765 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:52.823725    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerStarted","Data":"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"}
Sep 18 20:22:52.824252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:52.823793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerStarted","Data":"355658202d7cc345183093f26d34ad6d8ddc6b00ae7d9d149cbdf977a40f0065"}
Sep 18 20:22:55.681552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:55.681474    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:22:56.488356 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:56.488309    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:22:56.488526 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:56.488471    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:22:56.842230 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:56.842191    2573 generic.go:356] "Generic (PLEG): container finished" podID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerID="211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60" exitCode=0
Sep 18 20:22:56.842638 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:56.842240    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerDied","Data":"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"}
Sep 18 20:22:56.843289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:22:56.843273    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:23:01.488392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:01.488345    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:23:01.494391 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:01.494347    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:23:05.680887 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:05.680835    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:23:06.488669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:06.488623    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:23:08.900547 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.900514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerStarted","Data":"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"}
Sep 18 20:23:08.900954 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.900574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerStarted","Data":"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"}
Sep 18 20:23:08.900954 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.900826    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:23:08.900954 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.900873    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:23:08.902028 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.902003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:08.917652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:08.917603    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podStartSLOduration=2.192977727 podStartE2EDuration="17.917586662s" totalImagesPullingTime="11.091658416s" totalInitContainerRuntime="4.632950519s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:08.916766887 +0000 UTC m=+2697.434736528" watchObservedRunningTime="2026-09-18 20:23:08.917586662 +0000 UTC m=+2697.435556304"
Sep 18 20:23:09.904614 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:09.904566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:11.488970 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:11.488933    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:23:11.494376 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:11.494338    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:23:11.494543 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:11.494459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:23:14.908810 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:14.908779    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:23:14.909466 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:14.909433    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:15.681413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:15.681377    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:23:16.270760 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.270736    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:23:16.356138 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356106    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location\") pod \"07940e22-4a6d-4de4-93db-f279977c3388\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") "
Sep 18 20:23:16.356138 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356140    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") pod \"07940e22-4a6d-4de4-93db-f279977c3388\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") "
Sep 18 20:23:16.356373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config\") pod \"07940e22-4a6d-4de4-93db-f279977c3388\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") "
Sep 18 20:23:16.356373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356216    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr9g4\" (UniqueName: \"kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4\") pod \"07940e22-4a6d-4de4-93db-f279977c3388\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") "
Sep 18 20:23:16.356373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356265    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert\") pod \"07940e22-4a6d-4de4-93db-f279977c3388\" (UID: \"07940e22-4a6d-4de4-93db-f279977c3388\") "
Sep 18 20:23:16.356534 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356507    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07940e22-4a6d-4de4-93db-f279977c3388" (UID: "07940e22-4a6d-4de4-93db-f279977c3388"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:16.356641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356614    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07940e22-4a6d-4de4-93db-f279977c3388" (UID: "07940e22-4a6d-4de4-93db-f279977c3388"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:16.356697 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.356673    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07940e22-4a6d-4de4-93db-f279977c3388" (UID: "07940e22-4a6d-4de4-93db-f279977c3388"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:16.358345 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.358316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4" (OuterVolumeSpecName: "kube-api-access-hr9g4") pod "07940e22-4a6d-4de4-93db-f279977c3388" (UID: "07940e22-4a6d-4de4-93db-f279977c3388"). InnerVolumeSpecName "kube-api-access-hr9g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:16.358467 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.358440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07940e22-4a6d-4de4-93db-f279977c3388" (UID: "07940e22-4a6d-4de4-93db-f279977c3388"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:16.457864 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.457840    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07940e22-4a6d-4de4-93db-f279977c3388-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:16.457864 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.457861    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07940e22-4a6d-4de4-93db-f279977c3388-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:16.458025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.457870    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:16.458025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.457880    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr9g4\" (UniqueName: \"kubernetes.io/projected/07940e22-4a6d-4de4-93db-f279977c3388-kube-api-access-hr9g4\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:16.458025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.457889    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07940e22-4a6d-4de4-93db-f279977c3388-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:16.938019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.937974    2573 generic.go:356] "Generic (PLEG): container finished" podID="07940e22-4a6d-4de4-93db-f279977c3388" containerID="1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375" exitCode=137
Sep 18 20:23:16.938474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.938030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerDied","Data":"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"}
Sep 18 20:23:16.938474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.938054    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"
Sep 18 20:23:16.938474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.938115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn" event={"ID":"07940e22-4a6d-4de4-93db-f279977c3388","Type":"ContainerDied","Data":"3c0d4cb7c48e72fe8f75f9bcea183649b44c2ce8ab9b0e5570790f736312efe8"}
Sep 18 20:23:16.938474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.938161    2573 scope.go:117] "RemoveContainer" containerID="9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"
Sep 18 20:23:16.946708 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.946683    2573 scope.go:117] "RemoveContainer" containerID="1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"
Sep 18 20:23:16.954224 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.954190    2573 scope.go:117] "RemoveContainer" containerID="6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"
Sep 18 20:23:16.962146 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.962127    2573 scope.go:117] "RemoveContainer" containerID="12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"
Sep 18 20:23:16.962587 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.962567    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:23:16.967804 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.967783    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-78bfb5fcd9-djmmn"]
Sep 18 20:23:16.970276 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.970259    2573 scope.go:117] "RemoveContainer" containerID="9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"
Sep 18 20:23:16.970542 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:16.970523    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef\": container with ID starting with 9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef not found: ID does not exist" containerID="9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"
Sep 18 20:23:16.970626 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.970547    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef"} err="failed to get container status \"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef\": rpc error: code = NotFound desc = could not find container \"9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef\": container with ID starting with 9e69d4422a13069605ddf9df4e1b84448868f486a2c8143d61c99458f90afaef not found: ID does not exist"
Sep 18 20:23:16.970626 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.970563    2573 scope.go:117] "RemoveContainer" containerID="1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"
Sep 18 20:23:16.970801 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:16.970784    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375\": container with ID starting with 1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375 not found: ID does not exist" containerID="1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"
Sep 18 20:23:16.970841 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.970805    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375"} err="failed to get container status \"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375\": rpc error: code = NotFound desc = could not find container \"1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375\": container with ID starting with 1d36c0a3f4decccfee1aec333900eec44b8eb38682b587d2e5cd247b4f2dd375 not found: ID does not exist"
Sep 18 20:23:16.970841 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.970818    2573 scope.go:117] "RemoveContainer" containerID="6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"
Sep 18 20:23:16.971051 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:16.971036    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476\": container with ID starting with 6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476 not found: ID does not exist" containerID="6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"
Sep 18 20:23:16.971087 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.971056    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476"} err="failed to get container status \"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476\": rpc error: code = NotFound desc = could not find container \"6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476\": container with ID starting with 6aa3fbac0178716ed6ace947f40fad1c05b7f0cab54525d8b7b898f5508df476 not found: ID does not exist"
Sep 18 20:23:16.971087 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.971067    2573 scope.go:117] "RemoveContainer" containerID="12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"
Sep 18 20:23:16.971299 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:16.971279    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239\": container with ID starting with 12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239 not found: ID does not exist" containerID="12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"
Sep 18 20:23:16.971354 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:16.971304    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239"} err="failed to get container status \"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239\": rpc error: code = NotFound desc = could not find container \"12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239\": container with ID starting with 12b2630a33079c1761569e8d9e4031da18a27f909f7664d990fa7a4ae3639239 not found: ID does not exist"
Sep 18 20:23:17.782190 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:17.782159    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07940e22-4a6d-4de4-93db-f279977c3388" path="/var/lib/kubelet/pods/07940e22-4a6d-4de4-93db-f279977c3388/volumes"
Sep 18 20:23:24.909422 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:24.909382    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:25.681431 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:25.681394    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:23:34.778039 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:34.778006    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:23:34.909969 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:34.909935    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:36.503532 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:36.503499    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:23:36.503982 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:36.503786    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container" containerID="cri-o://4fbd1f949ca5b82d71a2f7f869fb6f3723eb85b2fd25afc284dd4df18d544dbb" gracePeriod=30
Sep 18 20:23:36.503982 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:36.503852    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kube-rbac-proxy" containerID="cri-o://9f69411000f967241de480332fec80527c329c3f001c858eb329547b74756f89" gracePeriod=30
Sep 18 20:23:37.022903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.022873    2573 generic.go:356] "Generic (PLEG): container finished" podID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerID="9f69411000f967241de480332fec80527c329c3f001c858eb329547b74756f89" exitCode=2
Sep 18 20:23:37.022903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.022911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerDied","Data":"9f69411000f967241de480332fec80527c329c3f001c858eb329547b74756f89"}
Sep 18 20:23:37.593245 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593188    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593526    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593537    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593546    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-agent"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593551    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-agent"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593563    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593570    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593584    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="storage-initializer"
Sep 18 20:23:37.593586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593589    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="storage-initializer"
Sep 18 20:23:37.593830 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593637    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kube-rbac-proxy"
Sep 18 20:23:37.593830 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593645    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-container"
Sep 18 20:23:37.593830 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.593651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07940e22-4a6d-4de4-93db-f279977c3388" containerName="kserve-agent"
Sep 18 20:23:37.615560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.615534    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:37.615718 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.615649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.617619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.617598    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:23:37.617740 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.617682    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 18 20:23:37.733693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.733655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.733890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.733760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptlk\" (UniqueName: \"kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.733890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.733838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.733890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.733876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.834930 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.834899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.835120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.834943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.835120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.835010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.835120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.835052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptlk\" (UniqueName: \"kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.835120 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:37.835061    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 20:23:37.835463 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:37.835134    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls podName:41663f40-4376-4605-805c-e50fbfffa59f nodeName:}" failed. No retries permitted until 2026-09-18 20:23:38.335114894 +0000 UTC m=+2726.853084519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" (UID: "41663f40-4376-4605-805c-e50fbfffa59f") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 18 20:23:37.835463 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.835395    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.835760 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.835734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:37.843280 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:37.843228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptlk\" (UniqueName: \"kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:38.338815 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:38.338719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:38.341139 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:38.341115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:38.525146 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:38.525103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:38.811884 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:38.806718    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:39.034496 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:39.034454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerStarted","Data":"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"}
Sep 18 20:23:39.034705 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:39.034530    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerStarted","Data":"adb60903a68e8fdf1d2dd8133e9c25bc6413d553419c48928d449ce93d53d001"}
Sep 18 20:23:40.676350 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:40.676308    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 18 20:23:41.047836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.047800    2573 generic.go:356] "Generic (PLEG): container finished" podID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerID="4fbd1f949ca5b82d71a2f7f869fb6f3723eb85b2fd25afc284dd4df18d544dbb" exitCode=0
Sep 18 20:23:41.047988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.047866    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerDied","Data":"4fbd1f949ca5b82d71a2f7f869fb6f3723eb85b2fd25afc284dd4df18d544dbb"}
Sep 18 20:23:41.140610 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.140589    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:23:41.262408 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config\") pod \"57e0b5fb-5064-42db-8fc6-f58edee5d764\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") "
Sep 18 20:23:41.262575 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262427    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert\") pod \"57e0b5fb-5064-42db-8fc6-f58edee5d764\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") "
Sep 18 20:23:41.262575 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262466    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location\") pod \"57e0b5fb-5064-42db-8fc6-f58edee5d764\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") "
Sep 18 20:23:41.262575 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262515    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rl5k\" (UniqueName: \"kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k\") pod \"57e0b5fb-5064-42db-8fc6-f58edee5d764\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") "
Sep 18 20:23:41.262575 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls\") pod \"57e0b5fb-5064-42db-8fc6-f58edee5d764\" (UID: \"57e0b5fb-5064-42db-8fc6-f58edee5d764\") "
Sep 18 20:23:41.262794 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57e0b5fb-5064-42db-8fc6-f58edee5d764" (UID: "57e0b5fb-5064-42db-8fc6-f58edee5d764"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:41.262843 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262782    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57e0b5fb-5064-42db-8fc6-f58edee5d764" (UID: "57e0b5fb-5064-42db-8fc6-f58edee5d764"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:41.262843 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.262758    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57e0b5fb-5064-42db-8fc6-f58edee5d764" (UID: "57e0b5fb-5064-42db-8fc6-f58edee5d764"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:41.264544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.264520    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57e0b5fb-5064-42db-8fc6-f58edee5d764" (UID: "57e0b5fb-5064-42db-8fc6-f58edee5d764"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:41.264646 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.264628    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k" (OuterVolumeSpecName: "kube-api-access-5rl5k") pod "57e0b5fb-5064-42db-8fc6-f58edee5d764" (UID: "57e0b5fb-5064-42db-8fc6-f58edee5d764"). InnerVolumeSpecName "kube-api-access-5rl5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:41.363393 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.363354    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rl5k\" (UniqueName: \"kubernetes.io/projected/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-api-access-5rl5k\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:41.363393 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.363388    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57e0b5fb-5064-42db-8fc6-f58edee5d764-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:41.363393 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.363400    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:41.363647 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.363412    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57e0b5fb-5064-42db-8fc6-f58edee5d764-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:41.363647 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:41.363424    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57e0b5fb-5064-42db-8fc6-f58edee5d764-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:42.054577 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.054486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl" event={"ID":"57e0b5fb-5064-42db-8fc6-f58edee5d764","Type":"ContainerDied","Data":"234e79deedefaa130cea86923bc0960d49944a0877c02366dce7cd938bcfb777"}
Sep 18 20:23:42.054577 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.054518    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"
Sep 18 20:23:42.054577 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.054576    2573 scope.go:117] "RemoveContainer" containerID="9f69411000f967241de480332fec80527c329c3f001c858eb329547b74756f89"
Sep 18 20:23:42.063023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.063003    2573 scope.go:117] "RemoveContainer" containerID="4fbd1f949ca5b82d71a2f7f869fb6f3723eb85b2fd25afc284dd4df18d544dbb"
Sep 18 20:23:42.070488 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.070456    2573 scope.go:117] "RemoveContainer" containerID="abc8c8154c9b020d350bb16547865f2b2d1846d5f2d4cb6fcf4801caa397af6d"
Sep 18 20:23:42.072277 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.072257    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:23:42.076219 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:42.076187    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-wbxnl"]
Sep 18 20:23:43.782682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:43.782644    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" path="/var/lib/kubelet/pods/57e0b5fb-5064-42db-8fc6-f58edee5d764/volumes"
Sep 18 20:23:44.910116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:44.910077    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:23:46.071924 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:46.071900    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/0.log"
Sep 18 20:23:46.072319 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:46.071934    2573 generic.go:356] "Generic (PLEG): container finished" podID="41663f40-4376-4605-805c-e50fbfffa59f" containerID="56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1" exitCode=1
Sep 18 20:23:46.072319 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:46.072007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerDied","Data":"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"}
Sep 18 20:23:47.078066 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:47.078036    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/0.log"
Sep 18 20:23:47.078536 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:47.078155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerStarted","Data":"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"}
Sep 18 20:23:47.572407 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:47.572376    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:48.081873 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.081834    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer" containerID="cri-o://56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612" gracePeriod=30
Sep 18 20:23:48.763851 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.763822    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764260    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kube-rbac-proxy"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764282    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kube-rbac-proxy"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764296    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="storage-initializer"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764305    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="storage-initializer"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764313    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764323    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764405    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kserve-container"
Sep 18 20:23:48.764513 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.764415    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e0b5fb-5064-42db-8fc6-f58edee5d764" containerName="kube-rbac-proxy"
Sep 18 20:23:48.766602 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.766582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.768990 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.768966    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 18 20:23:48.769115 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.769098    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:23:48.777411 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.777387    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:23:48.822282 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.822264    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/1.log"
Sep 18 20:23:48.822693 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.822675    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/0.log"
Sep 18 20:23:48.824108 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.824092    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:48.926944 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.926869    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ptlk\" (UniqueName: \"kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk\") pod \"41663f40-4376-4605-805c-e50fbfffa59f\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") "
Sep 18 20:23:48.926944 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.926929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config\") pod \"41663f40-4376-4605-805c-e50fbfffa59f\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") "
Sep 18 20:23:48.927160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927050    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location\") pod \"41663f40-4376-4605-805c-e50fbfffa59f\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") "
Sep 18 20:23:48.927160 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927093    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") pod \"41663f40-4376-4605-805c-e50fbfffa59f\" (UID: \"41663f40-4376-4605-805c-e50fbfffa59f\") "
Sep 18 20:23:48.927314 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927282    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41663f40-4376-4605-805c-e50fbfffa59f" (UID: "41663f40-4376-4605-805c-e50fbfffa59f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:23:48.927314 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.927433 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927326    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41663f40-4376-4605-805c-e50fbfffa59f" (UID: "41663f40-4376-4605-805c-e50fbfffa59f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:23:48.927433 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.927433 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.927650 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6qcn\" (UniqueName: \"kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.927650 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927527    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:48.927650 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927645    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41663f40-4376-4605-805c-e50fbfffa59f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:48.927801 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.927660    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41663f40-4376-4605-805c-e50fbfffa59f-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:48.929139 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.929114    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk" (OuterVolumeSpecName: "kube-api-access-8ptlk") pod "41663f40-4376-4605-805c-e50fbfffa59f" (UID: "41663f40-4376-4605-805c-e50fbfffa59f"). InnerVolumeSpecName "kube-api-access-8ptlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:23:48.929255 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:48.929210    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41663f40-4376-4605-805c-e50fbfffa59f" (UID: "41663f40-4376-4605-805c-e50fbfffa59f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:23:49.028331 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qcn\" (UniqueName: \"kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028607 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028461    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41663f40-4376-4605-805c-e50fbfffa59f-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:49.028607 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028477    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ptlk\" (UniqueName: \"kubernetes.io/projected/41663f40-4376-4605-805c-e50fbfffa59f-kube-api-access-8ptlk\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:23:49.028737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028715    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.028985 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.028959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.029094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.029035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.030658 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.030639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.036239 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.036177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qcn\" (UniqueName: \"kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.078034 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.078014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:49.088115 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088089    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/1.log"
Sep 18 20:23:49.088514 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088498    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg_41663f40-4376-4605-805c-e50fbfffa59f/storage-initializer/0.log"
Sep 18 20:23:49.088590 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088531    2573 generic.go:356] "Generic (PLEG): container finished" podID="41663f40-4376-4605-805c-e50fbfffa59f" containerID="56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612" exitCode=1
Sep 18 20:23:49.088590 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088583    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"
Sep 18 20:23:49.088701 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerDied","Data":"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"}
Sep 18 20:23:49.088754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg" event={"ID":"41663f40-4376-4605-805c-e50fbfffa59f","Type":"ContainerDied","Data":"adb60903a68e8fdf1d2dd8133e9c25bc6413d553419c48928d449ce93d53d001"}
Sep 18 20:23:49.088803 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.088772    2573 scope.go:117] "RemoveContainer" containerID="56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"
Sep 18 20:23:49.105276 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.105254    2573 scope.go:117] "RemoveContainer" containerID="56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"
Sep 18 20:23:49.112695 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.112676    2573 scope.go:117] "RemoveContainer" containerID="56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"
Sep 18 20:23:49.112962 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:49.112945    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612\": container with ID starting with 56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612 not found: ID does not exist" containerID="56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"
Sep 18 20:23:49.113017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.112972    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612"} err="failed to get container status \"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612\": rpc error: code = NotFound desc = could not find container \"56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612\": container with ID starting with 56a564c526de467390d856ebfa157e76697148dc6953dd809ec85751aacea612 not found: ID does not exist"
Sep 18 20:23:49.113017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.112989    2573 scope.go:117] "RemoveContainer" containerID="56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"
Sep 18 20:23:49.113271 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:23:49.113247    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1\": container with ID starting with 56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1 not found: ID does not exist" containerID="56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"
Sep 18 20:23:49.113327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.113281    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1"} err="failed to get container status \"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1\": rpc error: code = NotFound desc = could not find container \"56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1\": container with ID starting with 56daa6cbf6c63acf118517534a115b9afbacd236e52de6650b312cea195d1ad1 not found: ID does not exist"
Sep 18 20:23:49.130018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.129991    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:49.134443 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.134422    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-fm7xg"]
Sep 18 20:23:49.354664 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.354642    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:23:49.356938 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:23:49.356917    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb83a2a_6e90_4a7a_8ac0_aa8a70ee0311.slice/crio-bc2ce5d11773f0cd5212bb147c0254d757fa44ee6645c65edf7f0003548a15fa WatchSource:0}: Error finding container bc2ce5d11773f0cd5212bb147c0254d757fa44ee6645c65edf7f0003548a15fa: Status 404 returned error can't find the container with id bc2ce5d11773f0cd5212bb147c0254d757fa44ee6645c65edf7f0003548a15fa
Sep 18 20:23:49.783030 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:49.782953    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41663f40-4376-4605-805c-e50fbfffa59f" path="/var/lib/kubelet/pods/41663f40-4376-4605-805c-e50fbfffa59f/volumes"
Sep 18 20:23:50.095139 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:50.095106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerStarted","Data":"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"}
Sep 18 20:23:50.095573 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:50.095177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerStarted","Data":"bc2ce5d11773f0cd5212bb147c0254d757fa44ee6645c65edf7f0003548a15fa"}
Sep 18 20:23:51.102558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:51.102524    2573 generic.go:356] "Generic (PLEG): container finished" podID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerID="53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a" exitCode=0
Sep 18 20:23:51.102950 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:51.102587    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerDied","Data":"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"}
Sep 18 20:23:52.109704 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:52.109670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerStarted","Data":"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"}
Sep 18 20:23:52.110065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:52.109735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:52.110065 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:52.109756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerStarted","Data":"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"}
Sep 18 20:23:52.132271 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:52.132189    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podStartSLOduration=3.107684141 podStartE2EDuration="4.132169298s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.024485157s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:52.129194619 +0000 UTC m=+2740.647164273" watchObservedRunningTime="2026-09-18 20:23:52.132169298 +0000 UTC m=+2740.650138941"
Sep 18 20:23:53.114854 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:53.114828    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:53.116080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:53.116052    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:23:54.119225 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:54.119171    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:23:54.910368 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:54.910337    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:23:59.123802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:59.123774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:23:59.124294 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:23:59.124265    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:02.660862 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:02.660828    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:24:02.661357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:02.661134    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" containerID="cri-o://da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7" gracePeriod=30
Sep 18 20:24:02.661357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:02.661158    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kube-rbac-proxy" containerID="cri-o://4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f" gracePeriod=30
Sep 18 20:24:03.160022 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:03.159986    2573 generic.go:356] "Generic (PLEG): container finished" podID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerID="4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f" exitCode=2
Sep 18 20:24:03.160239 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:03.160059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerDied","Data":"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"}
Sep 18 20:24:04.905820 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:04.905774    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 18 20:24:04.910165 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:04.910139    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:24:05.996022 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:05.995998    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:24:06.070583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.070553    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2tnb\" (UniqueName: \"kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb\") pod \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") "
Sep 18 20:24:06.070728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.070602    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls\") pod \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") "
Sep 18 20:24:06.070728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.070656    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location\") pod \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") "
Sep 18 20:24:06.070728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.070677    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert\") pod \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") "
Sep 18 20:24:06.070728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.070702    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config\") pod \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\" (UID: \"07fe9711-2034-4f90-b6c0-fee97a58f2e6\") "
Sep 18 20:24:06.071074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.071043    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07fe9711-2034-4f90-b6c0-fee97a58f2e6" (UID: "07fe9711-2034-4f90-b6c0-fee97a58f2e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:06.071159 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.071069    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07fe9711-2034-4f90-b6c0-fee97a58f2e6" (UID: "07fe9711-2034-4f90-b6c0-fee97a58f2e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:24:06.072548 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.072526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb" (OuterVolumeSpecName: "kube-api-access-v2tnb") pod "07fe9711-2034-4f90-b6c0-fee97a58f2e6" (UID: "07fe9711-2034-4f90-b6c0-fee97a58f2e6"). InnerVolumeSpecName "kube-api-access-v2tnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:24:06.072660 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.072639    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07fe9711-2034-4f90-b6c0-fee97a58f2e6" (UID: "07fe9711-2034-4f90-b6c0-fee97a58f2e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:24:06.081760 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.081730    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07fe9711-2034-4f90-b6c0-fee97a58f2e6" (UID: "07fe9711-2034-4f90-b6c0-fee97a58f2e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:24:06.171617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.171557    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tnb\" (UniqueName: \"kubernetes.io/projected/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-api-access-v2tnb\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:24:06.171617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.171580    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fe9711-2034-4f90-b6c0-fee97a58f2e6-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:24:06.171617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.171589    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:24:06.171617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.171597    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:24:06.171617 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.171606    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07fe9711-2034-4f90-b6c0-fee97a58f2e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:24:06.175364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.175337    2573 generic.go:356] "Generic (PLEG): container finished" podID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerID="da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7" exitCode=0
Sep 18 20:24:06.175464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.175381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerDied","Data":"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"}
Sep 18 20:24:06.175464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.175404    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"
Sep 18 20:24:06.175464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.175452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8" event={"ID":"07fe9711-2034-4f90-b6c0-fee97a58f2e6","Type":"ContainerDied","Data":"355658202d7cc345183093f26d34ad6d8ddc6b00ae7d9d149cbdf977a40f0065"}
Sep 18 20:24:06.175574 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.175478    2573 scope.go:117] "RemoveContainer" containerID="4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"
Sep 18 20:24:06.184004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.183984    2573 scope.go:117] "RemoveContainer" containerID="da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"
Sep 18 20:24:06.191183 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.191169    2573 scope.go:117] "RemoveContainer" containerID="211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"
Sep 18 20:24:06.196407 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.196384    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:24:06.198803 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.198786    2573 scope.go:117] "RemoveContainer" containerID="4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"
Sep 18 20:24:06.199344 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:24:06.199291    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f\": container with ID starting with 4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f not found: ID does not exist" containerID="4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"
Sep 18 20:24:06.199544 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.199517    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f"} err="failed to get container status \"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f\": rpc error: code = NotFound desc = could not find container \"4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f\": container with ID starting with 4a73e0cac7f9106e4866d70acd96f9fab091ef9575976d0e352fc813f60d6d5f not found: ID does not exist"
Sep 18 20:24:06.199686 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.199671    2573 scope.go:117] "RemoveContainer" containerID="da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"
Sep 18 20:24:06.200082 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:24:06.200053    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7\": container with ID starting with da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7 not found: ID does not exist" containerID="da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"
Sep 18 20:24:06.200082 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.200077    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7"} err="failed to get container status \"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7\": rpc error: code = NotFound desc = could not find container \"da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7\": container with ID starting with da98d647e908f5d8418b707b584c476ec6b140e54a9f1e2107dee207e9e789c7 not found: ID does not exist"
Sep 18 20:24:06.200260 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.200092    2573 scope.go:117] "RemoveContainer" containerID="211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"
Sep 18 20:24:06.200436 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:24:06.200401    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60\": container with ID starting with 211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60 not found: ID does not exist" containerID="211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"
Sep 18 20:24:06.200508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.200444    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60"} err="failed to get container status \"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60\": rpc error: code = NotFound desc = could not find container \"211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60\": container with ID starting with 211f765500baaae422376ff0556513e72994f6d71227fa51cd5291cf784ace60 not found: ID does not exist"
Sep 18 20:24:06.201588 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:06.201567    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5c6v8"]
Sep 18 20:24:07.781976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.781947    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" path="/var/lib/kubelet/pods/07fe9711-2034-4f90-b6c0-fee97a58f2e6/volumes"
Sep 18 20:24:07.819488 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819457    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:24:07.819779 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819768    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819781    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819793    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="storage-initializer"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819798    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="storage-initializer"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819804    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kube-rbac-proxy"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819809    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kube-rbac-proxy"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819828    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container"
Sep 18 20:24:07.819836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819833    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819877    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kserve-container"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819885    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819891    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819897    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fe9711-2034-4f90-b6c0-fee97a58f2e6" containerName="kube-rbac-proxy"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819953    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.820053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.819959    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41663f40-4376-4605-805c-e50fbfffa59f" containerName="storage-initializer"
Sep 18 20:24:07.824295 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.824277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.826625 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.826602    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:24:07.826625 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.826617    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 18 20:24:07.826766 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.826611    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:24:07.832243 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.832219    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:24:07.885323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.885294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.885474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.885356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.885474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.885384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cnnb\" (UniqueName: \"kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.885474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.885424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.885474 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.885448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.986559 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.986700 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cnnb\" (UniqueName: \"kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.986700 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.986700 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.986700 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.987012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.986988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.987308 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.987277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.987308 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.987285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.989048 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.989020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:07.994634 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:07.994612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cnnb\" (UniqueName: \"kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb\") pod \"isvc-paddle-runtime-predictor-58b99fff76-2xpx4\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:08.137060 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:08.137030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:08.406281 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:08.406237    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:24:08.408384 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:24:08.408361    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32e0820_1a02_4196_89be_4a4a193050e8.slice/crio-022f9ba3addfcfb0e1cbb75c219e1479ad1105ecebdd7eb92d0ed363516988cf WatchSource:0}: Error finding container 022f9ba3addfcfb0e1cbb75c219e1479ad1105ecebdd7eb92d0ed363516988cf: Status 404 returned error can't find the container with id 022f9ba3addfcfb0e1cbb75c219e1479ad1105ecebdd7eb92d0ed363516988cf
Sep 18 20:24:09.124630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:09.124594    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:09.193505 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:09.193475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerStarted","Data":"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"}
Sep 18 20:24:09.193670 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:09.193529    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerStarted","Data":"022f9ba3addfcfb0e1cbb75c219e1479ad1105ecebdd7eb92d0ed363516988cf"}
Sep 18 20:24:13.212912 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:13.212880    2573 generic.go:356] "Generic (PLEG): container finished" podID="d32e0820-1a02-4196-89be-4a4a193050e8" containerID="f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5" exitCode=0
Sep 18 20:24:13.213307 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:13.212956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerDied","Data":"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"}
Sep 18 20:24:14.223800 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.223766    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerStarted","Data":"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"}
Sep 18 20:24:14.224152 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.223836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerStarted","Data":"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"}
Sep 18 20:24:14.224152 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.224101    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:14.224257 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.224224    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:14.225469 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.225437    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:14.242450 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:14.242403    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podStartSLOduration=2.681187178 podStartE2EDuration="7.242390258s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.56120308s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:14.240922162 +0000 UTC m=+2762.758891826" watchObservedRunningTime="2026-09-18 20:24:14.242390258 +0000 UTC m=+2762.760359898"
Sep 18 20:24:15.229175 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:15.229083    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:19.124736 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:19.124696    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:20.234403 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:20.234368    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:24:20.234967 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:20.234940    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:29.124874 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:29.124786    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:30.235792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:30.235754    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:39.125005 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:39.124964    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:40.234903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:40.234863    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:49.124962 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:49.124917    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:24:50.234825 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:50.234785    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:24:59.125186 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:24:59.125147    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:25:00.235749 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:00.235720    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:25:02.778023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:02.777979    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:25:09.233179 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:09.233141    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:25:09.233651 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:09.233504    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" containerID="cri-o://acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1" gracePeriod=30
Sep 18 20:25:09.233651 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:09.233607    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kube-rbac-proxy" containerID="cri-o://3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc" gracePeriod=30
Sep 18 20:25:09.454375 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:09.454326    2573 generic.go:356] "Generic (PLEG): container finished" podID="d32e0820-1a02-4196-89be-4a4a193050e8" containerID="3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc" exitCode=2
Sep 18 20:25:09.454537 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:09.454395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerDied","Data":"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"}
Sep 18 20:25:10.229353 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:10.229303    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 18 20:25:10.235551 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:10.235522    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:25:12.281534 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.281512    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:25:12.414026 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.413945    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location\") pod \"d32e0820-1a02-4196-89be-4a4a193050e8\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") "
Sep 18 20:25:12.414026 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414006    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config\") pod \"d32e0820-1a02-4196-89be-4a4a193050e8\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") "
Sep 18 20:25:12.414274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414062    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls\") pod \"d32e0820-1a02-4196-89be-4a4a193050e8\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") "
Sep 18 20:25:12.414274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414085    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cnnb\" (UniqueName: \"kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb\") pod \"d32e0820-1a02-4196-89be-4a4a193050e8\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") "
Sep 18 20:25:12.414274 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414137    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert\") pod \"d32e0820-1a02-4196-89be-4a4a193050e8\" (UID: \"d32e0820-1a02-4196-89be-4a4a193050e8\") "
Sep 18 20:25:12.414450 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414365    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d32e0820-1a02-4196-89be-4a4a193050e8" (UID: "d32e0820-1a02-4196-89be-4a4a193050e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:12.414565 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.414544    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d32e0820-1a02-4196-89be-4a4a193050e8" (UID: "d32e0820-1a02-4196-89be-4a4a193050e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:12.416347 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.416320    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb" (OuterVolumeSpecName: "kube-api-access-6cnnb") pod "d32e0820-1a02-4196-89be-4a4a193050e8" (UID: "d32e0820-1a02-4196-89be-4a4a193050e8"). InnerVolumeSpecName "kube-api-access-6cnnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:12.416454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.416379    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d32e0820-1a02-4196-89be-4a4a193050e8" (UID: "d32e0820-1a02-4196-89be-4a4a193050e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:12.422979 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.422956    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d32e0820-1a02-4196-89be-4a4a193050e8" (UID: "d32e0820-1a02-4196-89be-4a4a193050e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:12.468430 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.468397    2573 generic.go:356] "Generic (PLEG): container finished" podID="d32e0820-1a02-4196-89be-4a4a193050e8" containerID="acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1" exitCode=0
Sep 18 20:25:12.468583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.468441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerDied","Data":"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"}
Sep 18 20:25:12.468583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.468451    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"
Sep 18 20:25:12.468583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.468503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4" event={"ID":"d32e0820-1a02-4196-89be-4a4a193050e8","Type":"ContainerDied","Data":"022f9ba3addfcfb0e1cbb75c219e1479ad1105ecebdd7eb92d0ed363516988cf"}
Sep 18 20:25:12.468583 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.468543    2573 scope.go:117] "RemoveContainer" containerID="3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"
Sep 18 20:25:12.477245 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.477223    2573 scope.go:117] "RemoveContainer" containerID="acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"
Sep 18 20:25:12.484587 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.484572    2573 scope.go:117] "RemoveContainer" containerID="f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"
Sep 18 20:25:12.489381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.489360    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:25:12.492156 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.492142    2573 scope.go:117] "RemoveContainer" containerID="3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"
Sep 18 20:25:12.492501 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:12.492462    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc\": container with ID starting with 3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc not found: ID does not exist" containerID="3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"
Sep 18 20:25:12.492580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.492494    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc"} err="failed to get container status \"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc\": rpc error: code = NotFound desc = could not find container \"3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc\": container with ID starting with 3f76bf0b2f840b630e98cfec7cff640552f4fdd4d03d9ec0250d4912aa08b5dc not found: ID does not exist"
Sep 18 20:25:12.492580 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.492512    2573 scope.go:117] "RemoveContainer" containerID="acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"
Sep 18 20:25:12.492951 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:12.492829    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1\": container with ID starting with acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1 not found: ID does not exist" containerID="acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"
Sep 18 20:25:12.492951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.492895    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1"} err="failed to get container status \"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1\": rpc error: code = NotFound desc = could not find container \"acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1\": container with ID starting with acd9568415e8ed459485d37d7ab2c2e763077b4984bf65564cae281a31779dc1 not found: ID does not exist"
Sep 18 20:25:12.492951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.492940    2573 scope.go:117] "RemoveContainer" containerID="f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"
Sep 18 20:25:12.493489 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:12.493457    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5\": container with ID starting with f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5 not found: ID does not exist" containerID="f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"
Sep 18 20:25:12.493591 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.493538    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5"} err="failed to get container status \"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5\": rpc error: code = NotFound desc = could not find container \"f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5\": container with ID starting with f6977af51218da80f09463d081731094290d600cfb4dc3d1944a39677e6c22d5 not found: ID does not exist"
Sep 18 20:25:12.494974 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.494952    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-2xpx4"]
Sep 18 20:25:12.514934 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.514917    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d32e0820-1a02-4196-89be-4a4a193050e8-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.514934 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.514935    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cnnb\" (UniqueName: \"kubernetes.io/projected/d32e0820-1a02-4196-89be-4a4a193050e8-kube-api-access-6cnnb\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.515044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.514945    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.515044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.514955    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d32e0820-1a02-4196-89be-4a4a193050e8-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.515044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.514963    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d32e0820-1a02-4196-89be-4a4a193050e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:12.778449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:12.778363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:25:13.782465 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:13.782436    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" path="/var/lib/kubelet/pods/d32e0820-1a02-4196-89be-4a4a193050e8/volumes"
Sep 18 20:25:14.418133 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418099    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:25:14.418472 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418458    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container"
Sep 18 20:25:14.418524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418473    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container"
Sep 18 20:25:14.418524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418493    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="storage-initializer"
Sep 18 20:25:14.418524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="storage-initializer"
Sep 18 20:25:14.418524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418506    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kube-rbac-proxy"
Sep 18 20:25:14.418524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418512    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kube-rbac-proxy"
Sep 18 20:25:14.418679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418554    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kube-rbac-proxy"
Sep 18 20:25:14.418679 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.418561    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32e0820-1a02-4196-89be-4a4a193050e8" containerName="kserve-container"
Sep 18 20:25:14.421553 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.421534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.423696 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.423669    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 20:25:14.423820 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.423701    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:25:14.423820 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.423763    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 18 20:25:14.431777 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.431756    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:25:14.528630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.528599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.528806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.528638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.528806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.528689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.528806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.528765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.528806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.528791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbf8n\" (UniqueName: \"kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629290 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629253    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbf8n\" (UniqueName: \"kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.629673 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:14.629567    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 20:25:14.629673 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:14.629662    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls podName:89179503-6c8e-439c-917e-275d4824ffbc nodeName:}" failed. No retries permitted until 2026-09-18 20:25:15.129637646 +0000 UTC m=+2823.647607264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" (UID: "89179503-6c8e-439c-917e-275d4824ffbc") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 20:25:14.629784 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.630009 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.629990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.630059 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.630002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:14.637892 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:14.637857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbf8n\" (UniqueName: \"kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:15.133268 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:15.133222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:15.135866 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:15.135838    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:15.332162 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:15.332075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:15.601434 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:15.601408    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:25:15.604433 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:25:15.604410    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89179503_6c8e_439c_917e_275d4824ffbc.slice/crio-57ea98986c8609ad84877b76cc68c84d34cda5bba8ce17d9b92d25634c0d5f51 WatchSource:0}: Error finding container 57ea98986c8609ad84877b76cc68c84d34cda5bba8ce17d9b92d25634c0d5f51: Status 404 returned error can't find the container with id 57ea98986c8609ad84877b76cc68c84d34cda5bba8ce17d9b92d25634c0d5f51
Sep 18 20:25:16.487637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:16.487603    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerStarted","Data":"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"}
Sep 18 20:25:16.488054 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:16.487659    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerStarted","Data":"57ea98986c8609ad84877b76cc68c84d34cda5bba8ce17d9b92d25634c0d5f51"}
Sep 18 20:25:18.742490 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:18.742459    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:25:18.742852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:18.742700    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" containerID="cri-o://58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9" gracePeriod=30
Sep 18 20:25:18.742852 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:18.742747    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kube-rbac-proxy" containerID="cri-o://9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5" gracePeriod=30
Sep 18 20:25:19.119444 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.119405    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 20:25:19.502548 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.502467    2573 generic.go:356] "Generic (PLEG): container finished" podID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerID="9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5" exitCode=2
Sep 18 20:25:19.502548 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.502510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerDied","Data":"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"}
Sep 18 20:25:19.845357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.845324    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:19.851909 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.851379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:19.854342 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.854317    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 18 20:25:19.854342 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.854320    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:25:19.858090 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.858069    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:19.978104 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.978077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblqt\" (UniqueName: \"kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:19.978622 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.978603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:19.978716 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.978637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:19.978716 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:19.978666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.079991 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.079960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.079991 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.079993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.080237 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.080012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.080237 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.080037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblqt\" (UniqueName: \"kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.080437 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.080410    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.080697 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.080678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.082598 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.082579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.087739 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.087718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblqt\" (UniqueName: \"kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.167553 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.167473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:20.437728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.437703    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:20.439553 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:25:20.439527    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d8535a_24fa_4f82_aedc_4a959b539bf2.slice/crio-182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c WatchSource:0}: Error finding container 182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c: Status 404 returned error can't find the container with id 182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c
Sep 18 20:25:20.508783 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.508752    2573 generic.go:356] "Generic (PLEG): container finished" podID="89179503-6c8e-439c-917e-275d4824ffbc" containerID="9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7" exitCode=0
Sep 18 20:25:20.508940 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.508794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerDied","Data":"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"}
Sep 18 20:25:20.510572 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.510553    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerStarted","Data":"9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"}
Sep 18 20:25:20.510651 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:20.510596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerStarted","Data":"182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c"}
Sep 18 20:25:21.517102 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.517065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerStarted","Data":"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"}
Sep 18 20:25:21.517591 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.517134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerStarted","Data":"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"}
Sep 18 20:25:21.517591 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.517573    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:21.517729 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.517714    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:21.518989 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.518963    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:25:21.536352 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:21.536302    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podStartSLOduration=3.239018829 podStartE2EDuration="7.536285626s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.297266797s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:21.534464724 +0000 UTC m=+2830.052434378" watchObservedRunningTime="2026-09-18 20:25:21.536285626 +0000 UTC m=+2830.054255264"
Sep 18 20:25:22.520983 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:22.520938    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:25:22.778454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:22.778357    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:25:23.486684 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.486661    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:25:23.526731 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.526699    2573 generic.go:356] "Generic (PLEG): container finished" podID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerID="58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9" exitCode=0
Sep 18 20:25:23.527120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.526757    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"
Sep 18 20:25:23.527120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.526756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerDied","Data":"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"}
Sep 18 20:25:23.527120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.526813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb" event={"ID":"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311","Type":"ContainerDied","Data":"bc2ce5d11773f0cd5212bb147c0254d757fa44ee6645c65edf7f0003548a15fa"}
Sep 18 20:25:23.527120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.526842    2573 scope.go:117] "RemoveContainer" containerID="9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"
Sep 18 20:25:23.535089 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.535071    2573 scope.go:117] "RemoveContainer" containerID="58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"
Sep 18 20:25:23.542377 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.542359    2573 scope.go:117] "RemoveContainer" containerID="53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"
Sep 18 20:25:23.550951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.550924    2573 scope.go:117] "RemoveContainer" containerID="9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"
Sep 18 20:25:23.551275 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:23.551254    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5\": container with ID starting with 9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5 not found: ID does not exist" containerID="9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"
Sep 18 20:25:23.551372 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.551282    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5"} err="failed to get container status \"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5\": rpc error: code = NotFound desc = could not find container \"9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5\": container with ID starting with 9d9c0d7b2335ed87e608645bcab8d07ae03bd9afd7c0b06a2a74a72f2ad3f9b5 not found: ID does not exist"
Sep 18 20:25:23.551372 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.551299    2573 scope.go:117] "RemoveContainer" containerID="58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"
Sep 18 20:25:23.551573 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:23.551555    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9\": container with ID starting with 58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9 not found: ID does not exist" containerID="58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"
Sep 18 20:25:23.551612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.551586    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9"} err="failed to get container status \"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9\": rpc error: code = NotFound desc = could not find container \"58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9\": container with ID starting with 58502709dbc41eb10f6ee76c3b8949cd911c42976132e4b7800dc19f8769cbb9 not found: ID does not exist"
Sep 18 20:25:23.551612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.551601    2573 scope.go:117] "RemoveContainer" containerID="53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"
Sep 18 20:25:23.551875 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:23.551853    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a\": container with ID starting with 53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a not found: ID does not exist" containerID="53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"
Sep 18 20:25:23.551981 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.551877    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a"} err="failed to get container status \"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a\": rpc error: code = NotFound desc = could not find container \"53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a\": container with ID starting with 53881822e1353079298fea22182d77268e3aa591936958f0f429c3aa5601935a not found: ID does not exist"
Sep 18 20:25:23.612839 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.612809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config\") pod \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") "
Sep 18 20:25:23.613011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.612855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert\") pod \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") "
Sep 18 20:25:23.613011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.612898    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6qcn\" (UniqueName: \"kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn\") pod \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") "
Sep 18 20:25:23.613011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.612919    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls\") pod \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") "
Sep 18 20:25:23.613011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.612980    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location\") pod \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\" (UID: \"feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311\") "
Sep 18 20:25:23.613272 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.613244    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" (UID: "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:23.613366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.613343    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" (UID: "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:23.613476 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.613409    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" (UID: "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:23.615116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.615089    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" (UID: "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:23.615266 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.615122    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn" (OuterVolumeSpecName: "kube-api-access-m6qcn") pod "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" (UID: "feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311"). InnerVolumeSpecName "kube-api-access-m6qcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:23.713715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.713655    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.713715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.713715    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.713715 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.713731    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.713970 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.713745    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6qcn\" (UniqueName: \"kubernetes.io/projected/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-kube-api-access-m6qcn\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.713970 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.713759    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:23.858046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.858016    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:25:23.863708 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:23.863654    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-29lrb"]
Sep 18 20:25:24.535324 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:24.535298    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/0.log"
Sep 18 20:25:24.535817 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:24.535337    2573 generic.go:356] "Generic (PLEG): container finished" podID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerID="9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8" exitCode=1
Sep 18 20:25:24.535817 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:24.535419    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerDied","Data":"9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"}
Sep 18 20:25:25.541995 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:25.541967    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/0.log"
Sep 18 20:25:25.542420 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:25.542093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerStarted","Data":"b675caea13db110307edb07c14fb98ab0d980073d4e234a1c98492524a06bcf5"}
Sep 18 20:25:25.782899 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:25.782868    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" path="/var/lib/kubelet/pods/feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311/volumes"
Sep 18 20:25:27.526256 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:27.526225    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:25:27.526941 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:27.526903    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:25:29.560682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.560610    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/1.log"
Sep 18 20:25:29.561040 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.560981    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/0.log"
Sep 18 20:25:29.561040 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.561014    2573 generic.go:356] "Generic (PLEG): container finished" podID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerID="b675caea13db110307edb07c14fb98ab0d980073d4e234a1c98492524a06bcf5" exitCode=1
Sep 18 20:25:29.561118 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.561042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerDied","Data":"b675caea13db110307edb07c14fb98ab0d980073d4e234a1c98492524a06bcf5"}
Sep 18 20:25:29.561118 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.561103    2573 scope.go:117] "RemoveContainer" containerID="9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"
Sep 18 20:25:29.561498 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.561479    2573 scope.go:117] "RemoveContainer" containerID="9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"
Sep 18 20:25:29.572605 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:29.572582    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_kserve-ci-e2e-test(71d8535a-24fa-4f82-aedc-4a959b539bf2)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2"
Sep 18 20:25:29.572687 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:25:29.572597    2573 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_kserve-ci-e2e-test_71d8535a-24fa-4f82-aedc-4a959b539bf2_0 in pod sandbox 182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c from index: no such id: '9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8'" containerID="9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"
Sep 18 20:25:29.572687 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.572632    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_kserve-ci-e2e-test_71d8535a-24fa-4f82-aedc-4a959b539bf2_0 in pod sandbox 182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c from index: no such id: '9dfe8bf9d8dc254ace4c273b62881dd97e734f603ee5e46fb27c3cdc7c97edb8'"
Sep 18 20:25:29.827952 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:29.827854    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:30.566369 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.566334    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/1.log"
Sep 18 20:25:30.698277 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.698256    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/1.log"
Sep 18 20:25:30.703019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.703001    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:30.877903 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.877875    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mblqt\" (UniqueName: \"kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt\") pod \"71d8535a-24fa-4f82-aedc-4a959b539bf2\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") "
Sep 18 20:25:30.878092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.877938    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls\") pod \"71d8535a-24fa-4f82-aedc-4a959b539bf2\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") "
Sep 18 20:25:30.878092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.877966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config\") pod \"71d8535a-24fa-4f82-aedc-4a959b539bf2\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") "
Sep 18 20:25:30.878092 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.878019    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location\") pod \"71d8535a-24fa-4f82-aedc-4a959b539bf2\" (UID: \"71d8535a-24fa-4f82-aedc-4a959b539bf2\") "
Sep 18 20:25:30.878336 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.878306    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71d8535a-24fa-4f82-aedc-4a959b539bf2" (UID: "71d8535a-24fa-4f82-aedc-4a959b539bf2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:25:30.878336 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.878327    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71d8535a-24fa-4f82-aedc-4a959b539bf2" (UID: "71d8535a-24fa-4f82-aedc-4a959b539bf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:30.880081 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.880049    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71d8535a-24fa-4f82-aedc-4a959b539bf2" (UID: "71d8535a-24fa-4f82-aedc-4a959b539bf2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:30.880169 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.880128    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt" (OuterVolumeSpecName: "kube-api-access-mblqt") pod "71d8535a-24fa-4f82-aedc-4a959b539bf2" (UID: "71d8535a-24fa-4f82-aedc-4a959b539bf2"). InnerVolumeSpecName "kube-api-access-mblqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:30.949556 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949522    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:25:30.949871 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949860    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="storage-initializer"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949873    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="storage-initializer"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949882    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kube-rbac-proxy"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kube-rbac-proxy"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949901    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949907    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949913    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949918    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.949926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949927    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.950168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949933    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.950168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949981    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kserve-container"
Sep 18 20:25:30.950168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949988    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb83a2a-6e90-4a7a-8ac0-aa8a70ee0311" containerName="kube-rbac-proxy"
Sep 18 20:25:30.950168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.949996    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.950168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.950003    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" containerName="storage-initializer"
Sep 18 20:25:30.953841 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.953814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:30.955837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.955816    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 18 20:25:30.956072 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.956053    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:25:30.961963 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.961943    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:25:30.979402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.979368    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblqt\" (UniqueName: \"kubernetes.io/projected/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-api-access-mblqt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.979402 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.979394    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d8535a-24fa-4f82-aedc-4a959b539bf2-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.979566 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.979406    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d8535a-24fa-4f82-aedc-4a959b539bf2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:30.979566 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:30.979428    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d8535a-24fa-4f82-aedc-4a959b539bf2-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:25:31.079891 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.079859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.080074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.079927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.080074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.079948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.080074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.079965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84m7w\" (UniqueName: \"kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.080074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.079988    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181132 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181132 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181140    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m7w\" (UniqueName: \"kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181695 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181881 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.181946 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.181892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.183733 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.183714    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.188432 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.188411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84m7w\" (UniqueName: \"kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.264738 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.264702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:31.538926 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.538893    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:25:31.541777 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:25:31.541748    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf879f24_1d55_4162_ab68_74967b667767.slice/crio-7ec571d9a0fe37b3518c27899ccd7989c8e1c1511797fa8de546b03333c91ee2 WatchSource:0}: Error finding container 7ec571d9a0fe37b3518c27899ccd7989c8e1c1511797fa8de546b03333c91ee2: Status 404 returned error can't find the container with id 7ec571d9a0fe37b3518c27899ccd7989c8e1c1511797fa8de546b03333c91ee2
Sep 18 20:25:31.572273 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.572249    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx_71d8535a-24fa-4f82-aedc-4a959b539bf2/storage-initializer/1.log"
Sep 18 20:25:31.572703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.572345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx" event={"ID":"71d8535a-24fa-4f82-aedc-4a959b539bf2","Type":"ContainerDied","Data":"182689b5c621288426c370b7156dd17d3ba36704bd43c3677c1754310e82438c"}
Sep 18 20:25:31.572703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.572384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"
Sep 18 20:25:31.572703 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.572403    2573 scope.go:117] "RemoveContainer" containerID="b675caea13db110307edb07c14fb98ab0d980073d4e234a1c98492524a06bcf5"
Sep 18 20:25:31.573765 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.573689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerStarted","Data":"7ec571d9a0fe37b3518c27899ccd7989c8e1c1511797fa8de546b03333c91ee2"}
Sep 18 20:25:31.607545 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.607513    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:31.609737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.609709    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56d954cfb6-6fhtx"]
Sep 18 20:25:31.782745 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:31.782715    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d8535a-24fa-4f82-aedc-4a959b539bf2" path="/var/lib/kubelet/pods/71d8535a-24fa-4f82-aedc-4a959b539bf2/volumes"
Sep 18 20:25:32.580356 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:32.580317    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerStarted","Data":"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"}
Sep 18 20:25:33.586212 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:33.586174    2573 generic.go:356] "Generic (PLEG): container finished" podID="df879f24-1d55-4162-ab68-74967b667767" containerID="e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97" exitCode=0
Sep 18 20:25:33.586660 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:33.586258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerDied","Data":"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"}
Sep 18 20:25:34.595925 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.595890    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerStarted","Data":"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"}
Sep 18 20:25:34.596308 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.595948    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerStarted","Data":"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"}
Sep 18 20:25:34.596308 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.596131    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:34.596308 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.596172    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:34.597599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.597572    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:25:34.617630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:34.617569    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podStartSLOduration=3.6013629270000003 podStartE2EDuration="4.617550227s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.0161873s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:25:34.613584805 +0000 UTC m=+2843.131554448" watchObservedRunningTime="2026-09-18 20:25:34.617550227 +0000 UTC m=+2843.135519870"
Sep 18 20:25:35.600268 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:35.600224    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:25:37.527802 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:37.527764    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:25:40.604844 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:40.604815    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:25:40.605321 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:40.605298    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:25:47.527508 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:47.527462    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:25:50.606116 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:50.606071    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:25:57.527242 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:25:57.527142    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:26:00.605898 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:00.605857    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:26:07.528052 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:07.528008    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:26:10.605763 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:10.605722    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:26:16.045859 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:16.045827    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:26:16.046294 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:16.046062    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" containerID="cri-o://3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d" gracePeriod=30
Sep 18 20:26:16.046294 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:16.046133    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kube-rbac-proxy" containerID="cri-o://6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4" gracePeriod=30
Sep 18 20:26:16.772562 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:16.772533    2573 generic.go:356] "Generic (PLEG): container finished" podID="89179503-6c8e-439c-917e-275d4824ffbc" containerID="6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4" exitCode=2
Sep 18 20:26:16.772737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:16.772571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerDied","Data":"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"}
Sep 18 20:26:17.521756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:17.521706    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 18 20:26:17.526849 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:17.526809    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:26:18.989829 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:18.989801    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:26:19.103046 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103019    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config\") pod \"89179503-6c8e-439c-917e-275d4824ffbc\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") "
Sep 18 20:26:19.103288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103065    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert\") pod \"89179503-6c8e-439c-917e-275d4824ffbc\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") "
Sep 18 20:26:19.103288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbf8n\" (UniqueName: \"kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n\") pod \"89179503-6c8e-439c-917e-275d4824ffbc\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") "
Sep 18 20:26:19.103288 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103256    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location\") pod \"89179503-6c8e-439c-917e-275d4824ffbc\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") "
Sep 18 20:26:19.103478 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103315    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") pod \"89179503-6c8e-439c-917e-275d4824ffbc\" (UID: \"89179503-6c8e-439c-917e-275d4824ffbc\") "
Sep 18 20:26:19.103478 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103443    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89179503-6c8e-439c-917e-275d4824ffbc" (UID: "89179503-6c8e-439c-917e-275d4824ffbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:26:19.103478 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89179503-6c8e-439c-917e-275d4824ffbc" (UID: "89179503-6c8e-439c-917e-275d4824ffbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:26:19.103630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103566    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:19.103630 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.103585    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89179503-6c8e-439c-917e-275d4824ffbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:19.105178 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.105155    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n" (OuterVolumeSpecName: "kube-api-access-rbf8n") pod "89179503-6c8e-439c-917e-275d4824ffbc" (UID: "89179503-6c8e-439c-917e-275d4824ffbc"). InnerVolumeSpecName "kube-api-access-rbf8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:19.105417 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.105400    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89179503-6c8e-439c-917e-275d4824ffbc" (UID: "89179503-6c8e-439c-917e-275d4824ffbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:19.116886 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.116861    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89179503-6c8e-439c-917e-275d4824ffbc" (UID: "89179503-6c8e-439c-917e-275d4824ffbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:19.204553 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.204526    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbf8n\" (UniqueName: \"kubernetes.io/projected/89179503-6c8e-439c-917e-275d4824ffbc-kube-api-access-rbf8n\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:19.204553 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.204551    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89179503-6c8e-439c-917e-275d4824ffbc-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:19.204701 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.204562    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89179503-6c8e-439c-917e-275d4824ffbc-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:19.787616 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.787583    2573 generic.go:356] "Generic (PLEG): container finished" podID="89179503-6c8e-439c-917e-275d4824ffbc" containerID="3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d" exitCode=0
Sep 18 20:26:19.787616 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.787623    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerDied","Data":"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"}
Sep 18 20:26:19.787886 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.787647    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"
Sep 18 20:26:19.787886 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.787668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt" event={"ID":"89179503-6c8e-439c-917e-275d4824ffbc","Type":"ContainerDied","Data":"57ea98986c8609ad84877b76cc68c84d34cda5bba8ce17d9b92d25634c0d5f51"}
Sep 18 20:26:19.787886 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.787695    2573 scope.go:117] "RemoveContainer" containerID="6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"
Sep 18 20:26:19.796875 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.796856    2573 scope.go:117] "RemoveContainer" containerID="3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"
Sep 18 20:26:19.804344 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.804326    2573 scope.go:117] "RemoveContainer" containerID="9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"
Sep 18 20:26:19.805539 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.805519    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:26:19.809136 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.809114    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-76jvt"]
Sep 18 20:26:19.811752 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.811732    2573 scope.go:117] "RemoveContainer" containerID="6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"
Sep 18 20:26:19.812032 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:19.812013    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4\": container with ID starting with 6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4 not found: ID does not exist" containerID="6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"
Sep 18 20:26:19.812091 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.812039    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4"} err="failed to get container status \"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4\": rpc error: code = NotFound desc = could not find container \"6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4\": container with ID starting with 6fbf921e5b85a1df942e433ec7c435d2bcdc10a5bfbde5bd20fd7e7a8df972d4 not found: ID does not exist"
Sep 18 20:26:19.812091 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.812056    2573 scope.go:117] "RemoveContainer" containerID="3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"
Sep 18 20:26:19.812312 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:19.812294    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d\": container with ID starting with 3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d not found: ID does not exist" containerID="3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"
Sep 18 20:26:19.812366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.812317    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d"} err="failed to get container status \"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d\": rpc error: code = NotFound desc = could not find container \"3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d\": container with ID starting with 3727035c2854ff4af722dc3ce76559928da5cbd3868dffa0c81fa294c6a67d1d not found: ID does not exist"
Sep 18 20:26:19.812366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.812329    2573 scope.go:117] "RemoveContainer" containerID="9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"
Sep 18 20:26:19.812555 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:19.812537    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7\": container with ID starting with 9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7 not found: ID does not exist" containerID="9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"
Sep 18 20:26:19.812611 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:19.812563    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7"} err="failed to get container status \"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7\": rpc error: code = NotFound desc = could not find container \"9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7\": container with ID starting with 9be4483b60f9ba616ebc526a7eaea5336c7bd71e181cd0fd1c9ad0734b6c7de7 not found: ID does not exist"
Sep 18 20:26:20.605454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:20.605412    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:26:21.220635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220603    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:26:21.220936 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220924    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container"
Sep 18 20:26:21.220988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220937    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container"
Sep 18 20:26:21.220988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220951    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="storage-initializer"
Sep 18 20:26:21.220988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220957    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="storage-initializer"
Sep 18 20:26:21.220988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220967    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kube-rbac-proxy"
Sep 18 20:26:21.220988 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.220974    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kube-rbac-proxy"
Sep 18 20:26:21.221142 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.221019    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kube-rbac-proxy"
Sep 18 20:26:21.221142 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.221027    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89179503-6c8e-439c-917e-275d4824ffbc" containerName="kserve-container"
Sep 18 20:26:21.225636 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.225616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.227792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.227772    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 18 20:26:21.227899 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.227773    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:26:21.227899 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.227779    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 18 20:26:21.234567 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.234548    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:26:21.319250 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.319193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.319447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.319256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.319447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.319301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgrm\" (UniqueName: \"kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.319447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.319363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.319447 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.319390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420347 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420347 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420603 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420603 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420603 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgrm\" (UniqueName: \"kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.420810 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.420791    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.421050 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.421025    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.421151 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.421098    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.422917 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.422891    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.427647 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.427624    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgrm\" (UniqueName: \"kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm\") pod \"isvc-pmml-predictor-8674795f59-6tpkp\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.538232 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.538124    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:21.782357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.782315    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89179503-6c8e-439c-917e-275d4824ffbc" path="/var/lib/kubelet/pods/89179503-6c8e-439c-917e-275d4824ffbc/volumes"
Sep 18 20:26:21.809341 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:21.809306    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:26:21.811614 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:26:21.811590    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0c06d5_7838_4d09_8f68_356b8dd79427.slice/crio-af248f26a2ea906bfcee0cb1ab17a5b7c9b214432d986689dbd7a4074b93ebb3 WatchSource:0}: Error finding container af248f26a2ea906bfcee0cb1ab17a5b7c9b214432d986689dbd7a4074b93ebb3: Status 404 returned error can't find the container with id af248f26a2ea906bfcee0cb1ab17a5b7c9b214432d986689dbd7a4074b93ebb3
Sep 18 20:26:22.802604 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:22.802564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerStarted","Data":"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"}
Sep 18 20:26:22.802989 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:22.802663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerStarted","Data":"af248f26a2ea906bfcee0cb1ab17a5b7c9b214432d986689dbd7a4074b93ebb3"}
Sep 18 20:26:25.814701 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:25.814667    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerID="7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f" exitCode=0
Sep 18 20:26:25.815083 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:25.814738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerDied","Data":"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"}
Sep 18 20:26:30.605579 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:30.605537    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:26:33.862406 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:33.862370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerStarted","Data":"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"}
Sep 18 20:26:33.862775 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:33.862436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerStarted","Data":"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"}
Sep 18 20:26:33.862775 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:33.862615    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:33.879762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:33.879714    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podStartSLOduration=1.860195915 podStartE2EDuration="12.879702071s" totalImagesPullingTime="7.317801552s" totalInitContainerRuntime="3.701704604s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:33.878303618 +0000 UTC m=+2902.396273261" watchObservedRunningTime="2026-09-18 20:26:33.879702071 +0000 UTC m=+2902.397671712"
Sep 18 20:26:34.866866 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:34.866831    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:34.868221 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:34.868178    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:26:35.871837 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:35.871800    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:26:40.606021 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:40.605983    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 18 20:26:40.876511 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:40.876434    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:26:40.876923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:40.876898    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:26:50.606365 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:50.606328    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:26:50.877063 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:50.876981    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:26:50.947558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:50.947526    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:26:50.948558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:50.948327    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container" containerID="cri-o://128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06" gracePeriod=30
Sep 18 20:26:50.948558 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:50.948337    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kube-rbac-proxy" containerID="cri-o://5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f" gracePeriod=30
Sep 18 20:26:51.939569 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:51.939535    2573 generic.go:356] "Generic (PLEG): container finished" podID="df879f24-1d55-4162-ab68-74967b667767" containerID="5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f" exitCode=2
Sep 18 20:26:51.940058 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:51.939570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerDied","Data":"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"}
Sep 18 20:26:52.039330 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.039296    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:26:52.042888 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.042865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.044927 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.044901    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 18 20:26:52.045026 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.044905    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 18 20:26:52.051584 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.051559    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:26:52.178281 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.178242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.178464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.178301    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.178464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.178381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.178464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.178415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5jz\" (UniqueName: \"kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.279422 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.279343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.279422 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.279377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5jz\" (UniqueName: \"kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.279633 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.279506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.279633 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.279561    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.279928 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.279911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.280193 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.280171    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.281756 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.281735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.287504 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.287482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5jz\" (UniqueName: \"kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.353220 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.353179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:26:52.620808 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.620688    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:26:52.623034 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:26:52.623008    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cab85a_5df3_4ccf_a3ac_87df5fb7010f.slice/crio-6efd9d69cac6a0dc04e84274528b71e9b1880f9e827be3570e1328d4bdf7e91d WatchSource:0}: Error finding container 6efd9d69cac6a0dc04e84274528b71e9b1880f9e827be3570e1328d4bdf7e91d: Status 404 returned error can't find the container with id 6efd9d69cac6a0dc04e84274528b71e9b1880f9e827be3570e1328d4bdf7e91d
Sep 18 20:26:52.945321 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.945245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerStarted","Data":"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"}
Sep 18 20:26:52.945321 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:52.945300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerStarted","Data":"6efd9d69cac6a0dc04e84274528b71e9b1880f9e827be3570e1328d4bdf7e91d"}
Sep 18 20:26:55.492986 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.492961    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:26:55.607547 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.607522    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84m7w\" (UniqueName: \"kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w\") pod \"df879f24-1d55-4162-ab68-74967b667767\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") "
Sep 18 20:26:55.607666 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.607579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls\") pod \"df879f24-1d55-4162-ab68-74967b667767\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") "
Sep 18 20:26:55.607666 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.607605    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert\") pod \"df879f24-1d55-4162-ab68-74967b667767\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") "
Sep 18 20:26:55.607666 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.607652    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location\") pod \"df879f24-1d55-4162-ab68-74967b667767\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") "
Sep 18 20:26:55.607850 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.607699    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config\") pod \"df879f24-1d55-4162-ab68-74967b667767\" (UID: \"df879f24-1d55-4162-ab68-74967b667767\") "
Sep 18 20:26:55.608044 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.608020    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df879f24-1d55-4162-ab68-74967b667767" (UID: "df879f24-1d55-4162-ab68-74967b667767"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:26:55.608109 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.608035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df879f24-1d55-4162-ab68-74967b667767" (UID: "df879f24-1d55-4162-ab68-74967b667767"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:26:55.608109 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.608090    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df879f24-1d55-4162-ab68-74967b667767" (UID: "df879f24-1d55-4162-ab68-74967b667767"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:26:55.609590 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.609573    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w" (OuterVolumeSpecName: "kube-api-access-84m7w") pod "df879f24-1d55-4162-ab68-74967b667767" (UID: "df879f24-1d55-4162-ab68-74967b667767"). InnerVolumeSpecName "kube-api-access-84m7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:26:55.609725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.609698    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df879f24-1d55-4162-ab68-74967b667767" (UID: "df879f24-1d55-4162-ab68-74967b667767"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:26:55.714304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.709485    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:55.714304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.709525    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84m7w\" (UniqueName: \"kubernetes.io/projected/df879f24-1d55-4162-ab68-74967b667767-kube-api-access-84m7w\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:55.714304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.709543    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df879f24-1d55-4162-ab68-74967b667767-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:55.714304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.709559    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df879f24-1d55-4162-ab68-74967b667767-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:55.714304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.709580    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df879f24-1d55-4162-ab68-74967b667767-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:26:55.961697 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.961666    2573 generic.go:356] "Generic (PLEG): container finished" podID="df879f24-1d55-4162-ab68-74967b667767" containerID="128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06" exitCode=0
Sep 18 20:26:55.961831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.961701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerDied","Data":"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"}
Sep 18 20:26:55.961831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.961733    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"
Sep 18 20:26:55.961831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.961762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm" event={"ID":"df879f24-1d55-4162-ab68-74967b667767","Type":"ContainerDied","Data":"7ec571d9a0fe37b3518c27899ccd7989c8e1c1511797fa8de546b03333c91ee2"}
Sep 18 20:26:55.961831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.961788    2573 scope.go:117] "RemoveContainer" containerID="5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"
Sep 18 20:26:55.971123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.971107    2573 scope.go:117] "RemoveContainer" containerID="128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"
Sep 18 20:26:55.980215 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.980172    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:26:55.981585 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:55.981567    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5694f8f689-kj4nm"]
Sep 18 20:26:56.021037 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.021019    2573 scope.go:117] "RemoveContainer" containerID="e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"
Sep 18 20:26:56.029523 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.029503    2573 scope.go:117] "RemoveContainer" containerID="5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"
Sep 18 20:26:56.029778 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:56.029759    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f\": container with ID starting with 5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f not found: ID does not exist" containerID="5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"
Sep 18 20:26:56.029853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.029787    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f"} err="failed to get container status \"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f\": rpc error: code = NotFound desc = could not find container \"5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f\": container with ID starting with 5c10fbc06f70db59d65c3656b63983366e0a9ccea5e8d8015fa85e2f38c91e5f not found: ID does not exist"
Sep 18 20:26:56.029853 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.029805    2573 scope.go:117] "RemoveContainer" containerID="128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"
Sep 18 20:26:56.030067 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:56.030053    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06\": container with ID starting with 128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06 not found: ID does not exist" containerID="128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"
Sep 18 20:26:56.030120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.030071    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06"} err="failed to get container status \"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06\": rpc error: code = NotFound desc = could not find container \"128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06\": container with ID starting with 128dd36eab98eb7ff2fd4e3a6bc41e463ece2c03b2451c48f669418fb8c8dd06 not found: ID does not exist"
Sep 18 20:26:56.030120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.030084    2573 scope.go:117] "RemoveContainer" containerID="e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"
Sep 18 20:26:56.030405 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:26:56.030387    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97\": container with ID starting with e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97 not found: ID does not exist" containerID="e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"
Sep 18 20:26:56.030449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.030414    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97"} err="failed to get container status \"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97\": rpc error: code = NotFound desc = could not find container \"e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97\": container with ID starting with e2155f8be2e80440331b584d4528df5e55c64030eced00b58b92ff49f7999a97 not found: ID does not exist"
Sep 18 20:26:56.968612 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.968588    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/0.log"
Sep 18 20:26:56.969068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.968619    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerID="66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5" exitCode=1
Sep 18 20:26:56.969068 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:56.968705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerDied","Data":"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"}
Sep 18 20:26:57.782299 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:57.782271    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df879f24-1d55-4162-ab68-74967b667767" path="/var/lib/kubelet/pods/df879f24-1d55-4162-ab68-74967b667767/volumes"
Sep 18 20:26:57.975018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:57.974990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/0.log"
Sep 18 20:26:57.975509 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:26:57.975090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerStarted","Data":"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"}
Sep 18 20:27:00.877728 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:00.877690    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:02.024273 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.024244    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:27:02.024646 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.024454    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer" containerID="cri-o://83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade" gracePeriod=30
Sep 18 20:27:02.459021 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.458999    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/1.log"
Sep 18 20:27:02.459421 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.459407    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/0.log"
Sep 18 20:27:02.461850 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.461836    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:27:02.569289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569184    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq5jz\" (UniqueName: \"kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz\") pod \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") "
Sep 18 20:27:02.569423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569307    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config\") pod \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") "
Sep 18 20:27:02.569423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location\") pod \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") "
Sep 18 20:27:02.569423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569402    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls\") pod \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\" (UID: \"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f\") "
Sep 18 20:27:02.569697 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569662    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" (UID: "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:27:02.569697 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.569673    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" (UID: "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:27:02.571392 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.571361    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz" (OuterVolumeSpecName: "kube-api-access-wq5jz") pod "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" (UID: "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f"). InnerVolumeSpecName "kube-api-access-wq5jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:27:02.571475 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.571422    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" (UID: "f3cab85a-5df3-4ccf-a3ac-87df5fb7010f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:27:02.670774 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.670740    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:27:02.670774 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.670767    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:27:02.670774 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.670777    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5jz\" (UniqueName: \"kubernetes.io/projected/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-api-access-wq5jz\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:27:02.671009 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.670786    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:27:02.997531 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.997506    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/1.log"
Sep 18 20:27:02.997928 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.997913    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n_f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/storage-initializer/0.log"
Sep 18 20:27:02.998011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.997951    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerID="83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade" exitCode=1
Sep 18 20:27:02.998011 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.997988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerDied","Data":"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"}
Sep 18 20:27:02.998127 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.998017    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"
Sep 18 20:27:02.998127 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.998071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n" event={"ID":"f3cab85a-5df3-4ccf-a3ac-87df5fb7010f","Type":"ContainerDied","Data":"6efd9d69cac6a0dc04e84274528b71e9b1880f9e827be3570e1328d4bdf7e91d"}
Sep 18 20:27:02.998127 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:02.998116    2573 scope.go:117] "RemoveContainer" containerID="83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"
Sep 18 20:27:03.006834 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.006820    2573 scope.go:117] "RemoveContainer" containerID="66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"
Sep 18 20:27:03.013997 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.013981    2573 scope.go:117] "RemoveContainer" containerID="83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"
Sep 18 20:27:03.014371 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:27:03.014347    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade\": container with ID starting with 83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade not found: ID does not exist" containerID="83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"
Sep 18 20:27:03.014454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.014377    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade"} err="failed to get container status \"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade\": rpc error: code = NotFound desc = could not find container \"83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade\": container with ID starting with 83e13d2758f4ac60e8f39efe876a1985a61b7702936a3b256c27a01e83ee7ade not found: ID does not exist"
Sep 18 20:27:03.014454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.014391    2573 scope.go:117] "RemoveContainer" containerID="66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"
Sep 18 20:27:03.014733 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:27:03.014711    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5\": container with ID starting with 66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5 not found: ID does not exist" containerID="66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"
Sep 18 20:27:03.014789 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.014743    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5"} err="failed to get container status \"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5\": rpc error: code = NotFound desc = could not find container \"66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5\": container with ID starting with 66510947b724c7348957cdd99138f0600241281863fa120112ae70354b75cca5 not found: ID does not exist"
Sep 18 20:27:03.036921 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.036895    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:27:03.054261 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.054236    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c7767bfc5-px85n"]
Sep 18 20:27:03.196324 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196295    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:27:03.196606 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196595    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.196654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196608    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.196654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196617    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="storage-initializer"
Sep 18 20:27:03.196654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196623    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="storage-initializer"
Sep 18 20:27:03.196654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196635    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kube-rbac-proxy"
Sep 18 20:27:03.196654 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196641    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kube-rbac-proxy"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196659    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196664    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196707    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196715    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kserve-container"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196722    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df879f24-1d55-4162-ab68-74967b667767" containerName="kube-rbac-proxy"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196780    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.196806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196785    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.197018 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.196826    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" containerName="storage-initializer"
Sep 18 20:27:03.201443 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.201421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.203689 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.203669    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 18 20:27:03.203811 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.203722    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 20:27:03.210000 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.209978    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:27:03.275834 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.275759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.275834 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.275799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.276008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.275851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.276008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.275891    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.276008 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.275973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fxv\" (UniqueName: \"kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376581 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.376550    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376726 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.376598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fxv\" (UniqueName: \"kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376726 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.376637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376726 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.376669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376726 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.376700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.376919 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:27:03.376825    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 18 20:27:03.376919 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:27:03.376905    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls podName:0d487173-29cf-4a30-9026-e217427195c3 nodeName:}" failed. No retries permitted until 2026-09-18 20:27:03.876884537 +0000 UTC m=+2932.394854155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" (UID: "0d487173-29cf-4a30-9026-e217427195c3") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 18 20:27:03.377075 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.377025    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.377353 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.377337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.377413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.377349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.386854 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.386831    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fxv\" (UniqueName: \"kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.783234 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.783183    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cab85a-5df3-4ccf-a3ac-87df5fb7010f" path="/var/lib/kubelet/pods/f3cab85a-5df3-4ccf-a3ac-87df5fb7010f/volumes"
Sep 18 20:27:03.880806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.880772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:03.883125 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:03.883097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:04.112148 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:04.112114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:04.418247 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:04.418223    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:27:04.420811 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:27:04.420786    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d487173_29cf_4a30_9026_e217427195c3.slice/crio-b79503a84f78f578a33485ce982fdad08e5e9fa78e3a8fa93835dd92cc6c86f6 WatchSource:0}: Error finding container b79503a84f78f578a33485ce982fdad08e5e9fa78e3a8fa93835dd92cc6c86f6: Status 404 returned error can't find the container with id b79503a84f78f578a33485ce982fdad08e5e9fa78e3a8fa93835dd92cc6c86f6
Sep 18 20:27:05.010007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:05.009967    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerStarted","Data":"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"}
Sep 18 20:27:05.010178 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:05.010037    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerStarted","Data":"b79503a84f78f578a33485ce982fdad08e5e9fa78e3a8fa93835dd92cc6c86f6"}
Sep 18 20:27:09.027252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:09.027216    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d487173-29cf-4a30-9026-e217427195c3" containerID="00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017" exitCode=0
Sep 18 20:27:09.027613 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:09.027290    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerDied","Data":"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"}
Sep 18 20:27:10.034435 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.034392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerStarted","Data":"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"}
Sep 18 20:27:10.034914 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.034455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerStarted","Data":"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"}
Sep 18 20:27:10.034914 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.034766    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:10.034914 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.034892    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:10.036135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.036109    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:10.053446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.053399    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podStartSLOduration=3.23032302 podStartE2EDuration="7.053385426s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823062406s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:27:10.052472321 +0000 UTC m=+2938.570441974" watchObservedRunningTime="2026-09-18 20:27:10.053385426 +0000 UTC m=+2938.571355112"
Sep 18 20:27:10.876842 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:10.876807    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:11.038730 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:11.038694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:16.042740 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:16.042710    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:27:16.043256 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:16.043227    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:20.877920 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:20.877880    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:26.043744 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:26.043660    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:30.877243 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:30.877185    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:36.043682 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:36.043643    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:40.877357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:40.877316    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:46.043843 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:46.043807    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:27:50.877147 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:50.877107    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 18 20:27:56.043967 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:27:56.043925    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:00.877867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:00.877834    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:28:02.756799 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:02.756760    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:28:02.757248 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:02.757081    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container" containerID="cri-o://c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544" gracePeriod=30
Sep 18 20:28:02.757248 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:02.757129    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kube-rbac-proxy" containerID="cri-o://c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e" gracePeriod=30
Sep 18 20:28:03.247851 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:03.247818    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerID="c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e" exitCode=2
Sep 18 20:28:03.248016 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:03.247857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerDied","Data":"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"}
Sep 18 20:28:05.872571 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:05.872535    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Sep 18 20:28:06.044031 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.043994    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:06.899796 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.899771    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:28:06.995273 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995141    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert\") pod \"ad0c06d5-7838-4d09-8f68-356b8dd79427\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") "
Sep 18 20:28:06.995273 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995190    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls\") pod \"ad0c06d5-7838-4d09-8f68-356b8dd79427\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") "
Sep 18 20:28:06.995273 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995240    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbgrm\" (UniqueName: \"kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm\") pod \"ad0c06d5-7838-4d09-8f68-356b8dd79427\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") "
Sep 18 20:28:06.995568 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995288    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config\") pod \"ad0c06d5-7838-4d09-8f68-356b8dd79427\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") "
Sep 18 20:28:06.995568 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995323    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location\") pod \"ad0c06d5-7838-4d09-8f68-356b8dd79427\" (UID: \"ad0c06d5-7838-4d09-8f68-356b8dd79427\") "
Sep 18 20:28:06.995683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995612    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad0c06d5-7838-4d09-8f68-356b8dd79427" (UID: "ad0c06d5-7838-4d09-8f68-356b8dd79427"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:28:06.995683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995623    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad0c06d5-7838-4d09-8f68-356b8dd79427" (UID: "ad0c06d5-7838-4d09-8f68-356b8dd79427"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:28:06.995768 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.995674    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad0c06d5-7838-4d09-8f68-356b8dd79427" (UID: "ad0c06d5-7838-4d09-8f68-356b8dd79427"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:28:06.997469 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.997449    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad0c06d5-7838-4d09-8f68-356b8dd79427" (UID: "ad0c06d5-7838-4d09-8f68-356b8dd79427"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:28:06.997552 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:06.997526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm" (OuterVolumeSpecName: "kube-api-access-zbgrm") pod "ad0c06d5-7838-4d09-8f68-356b8dd79427" (UID: "ad0c06d5-7838-4d09-8f68-356b8dd79427"). InnerVolumeSpecName "kube-api-access-zbgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:28:07.096669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.096622    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.096669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.096663    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad0c06d5-7838-4d09-8f68-356b8dd79427-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.096669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.096673    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgrm\" (UniqueName: \"kubernetes.io/projected/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-api-access-zbgrm\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.096669 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.096682    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad0c06d5-7838-4d09-8f68-356b8dd79427-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.096949 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.096692    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad0c06d5-7838-4d09-8f68-356b8dd79427-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:07.266275 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.266166    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerID="c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544" exitCode=0
Sep 18 20:28:07.266275 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.266255    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"
Sep 18 20:28:07.266464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.266251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerDied","Data":"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"}
Sep 18 20:28:07.266464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.266388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp" event={"ID":"ad0c06d5-7838-4d09-8f68-356b8dd79427","Type":"ContainerDied","Data":"af248f26a2ea906bfcee0cb1ab17a5b7c9b214432d986689dbd7a4074b93ebb3"}
Sep 18 20:28:07.266464 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.266430    2573 scope.go:117] "RemoveContainer" containerID="c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"
Sep 18 20:28:07.275052 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.275035    2573 scope.go:117] "RemoveContainer" containerID="c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"
Sep 18 20:28:07.282265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.282248    2573 scope.go:117] "RemoveContainer" containerID="7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"
Sep 18 20:28:07.287493 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.287468    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:28:07.290579 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.290558    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-6tpkp"]
Sep 18 20:28:07.290645 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.290566    2573 scope.go:117] "RemoveContainer" containerID="c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"
Sep 18 20:28:07.290874 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:07.290855    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e\": container with ID starting with c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e not found: ID does not exist" containerID="c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"
Sep 18 20:28:07.290936 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.290885    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e"} err="failed to get container status \"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e\": rpc error: code = NotFound desc = could not find container \"c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e\": container with ID starting with c53fd22f81537ed52a69ac72335a06afde53b6ddd612f4e26994c1b7663b890e not found: ID does not exist"
Sep 18 20:28:07.290936 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.290907    2573 scope.go:117] "RemoveContainer" containerID="c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"
Sep 18 20:28:07.291157 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:07.291141    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544\": container with ID starting with c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544 not found: ID does not exist" containerID="c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"
Sep 18 20:28:07.291212 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.291162    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544"} err="failed to get container status \"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544\": rpc error: code = NotFound desc = could not find container \"c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544\": container with ID starting with c0a6918c6091f71a5199d1d81df10416f3531bc51a66e0eee76be4bf83928544 not found: ID does not exist"
Sep 18 20:28:07.291212 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.291178    2573 scope.go:117] "RemoveContainer" containerID="7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"
Sep 18 20:28:07.291407 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:07.291392    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f\": container with ID starting with 7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f not found: ID does not exist" containerID="7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"
Sep 18 20:28:07.291453 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.291411    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f"} err="failed to get container status \"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f\": rpc error: code = NotFound desc = could not find container \"7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f\": container with ID starting with 7c6804fdc93e654057553a37665de8326d9b44e5a68856d621bfcf888c53166f not found: ID does not exist"
Sep 18 20:28:07.782322 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.782292    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" path="/var/lib/kubelet/pods/ad0c06d5-7838-4d09-8f68-356b8dd79427/volumes"
Sep 18 20:28:07.924970 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.924934    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925264    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="storage-initializer"
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925280    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="storage-initializer"
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925309    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kube-rbac-proxy"
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925319    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kube-rbac-proxy"
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925338    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container"
Sep 18 20:28:07.925367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925347    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container"
Sep 18 20:28:07.925566 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925448    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kube-rbac-proxy"
Sep 18 20:28:07.925566 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.925461    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0c06d5-7838-4d09-8f68-356b8dd79427" containerName="kserve-container"
Sep 18 20:28:07.930676 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.930653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:07.932867 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.932850    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 18 20:28:07.932956 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.932889    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:28:07.939889 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:07.939863    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:28:08.004769 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.004738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.004942 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.004783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.004942 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.004804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.004942 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.004910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrlj\" (UniqueName: \"kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.004942 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.004937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.105843 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.105811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.105857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.105876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.105910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrlj\" (UniqueName: \"kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106012 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.105927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106452 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.106423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106631 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.106615    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.106676 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.106635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.108356 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.108337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.114080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.114058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrlj\" (UniqueName: \"kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj\") pod \"isvc-pmml-runtime-predictor-885fc8777-hvx2w\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.242639 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.242588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:08.514367 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:08.514278    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:28:08.517182 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:28:08.517157    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf958e977_bf39_45e4_a6b0_c16ab7018822.slice/crio-73aca87999d0abab4ccb2273cec176bc2b3636f78d148d5446d2402538fd4de0 WatchSource:0}: Error finding container 73aca87999d0abab4ccb2273cec176bc2b3636f78d148d5446d2402538fd4de0: Status 404 returned error can't find the container with id 73aca87999d0abab4ccb2273cec176bc2b3636f78d148d5446d2402538fd4de0
Sep 18 20:28:09.277417 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:09.277381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerStarted","Data":"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"}
Sep 18 20:28:09.277809 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:09.277454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerStarted","Data":"73aca87999d0abab4ccb2273cec176bc2b3636f78d148d5446d2402538fd4de0"}
Sep 18 20:28:13.295995 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:13.295957    2573 generic.go:356] "Generic (PLEG): container finished" podID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerID="a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382" exitCode=0
Sep 18 20:28:13.296420 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:13.296014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerDied","Data":"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"}
Sep 18 20:28:14.302866 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:14.302830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerStarted","Data":"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"}
Sep 18 20:28:14.303252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:14.302895    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerStarted","Data":"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"}
Sep 18 20:28:14.303252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:14.303104    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:14.320599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:14.320554    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podStartSLOduration=3.4923561149999998 podStartE2EDuration="7.320541042s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828184927s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:28:14.319490702 +0000 UTC m=+3002.837460343" watchObservedRunningTime="2026-09-18 20:28:14.320541042 +0000 UTC m=+3002.838510681"
Sep 18 20:28:15.307485 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:15.307402    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:15.308759 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:15.308730    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:16.043641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:16.043603    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:16.311286 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:16.311180    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:21.316017 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:21.315990    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:28:21.316632 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:21.316505    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:25.778060 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:25.778013    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:31.317096 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:31.317054    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:35.778149 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:35.778106    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:41.316906 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:41.316862    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:45.783836 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:45.783803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:28:51.316462 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:51.316427    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:28:55.232353 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.232268    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:28:55.232923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.232514    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" containerID="cri-o://1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525" gracePeriod=30
Sep 18 20:28:55.232923 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.232543    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kube-rbac-proxy" containerID="cri-o://1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3" gracePeriod=30
Sep 18 20:28:55.468832 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.468800    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d487173-29cf-4a30-9026-e217427195c3" containerID="1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3" exitCode=2
Sep 18 20:28:55.468994 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.468850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerDied","Data":"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"}
Sep 18 20:28:55.778948 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:55.778900    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 18 20:28:56.038974 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:56.038879    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Sep 18 20:28:59.475792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.475771    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:28:59.486673 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.486649    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d487173-29cf-4a30-9026-e217427195c3" containerID="1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525" exitCode=0
Sep 18 20:28:59.486793 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.486695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerDied","Data":"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"}
Sep 18 20:28:59.486793 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.486698    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"
Sep 18 20:28:59.486793 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.486742    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz" event={"ID":"0d487173-29cf-4a30-9026-e217427195c3","Type":"ContainerDied","Data":"b79503a84f78f578a33485ce982fdad08e5e9fa78e3a8fa93835dd92cc6c86f6"}
Sep 18 20:28:59.486793 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.486767    2573 scope.go:117] "RemoveContainer" containerID="1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"
Sep 18 20:28:59.494366 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.494333    2573 scope.go:117] "RemoveContainer" containerID="1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"
Sep 18 20:28:59.503187 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.503167    2573 scope.go:117] "RemoveContainer" containerID="00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"
Sep 18 20:28:59.510605 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.510583    2573 scope.go:117] "RemoveContainer" containerID="1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"
Sep 18 20:28:59.510891 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:59.510872    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3\": container with ID starting with 1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3 not found: ID does not exist" containerID="1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"
Sep 18 20:28:59.510959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.510901    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3"} err="failed to get container status \"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3\": rpc error: code = NotFound desc = could not find container \"1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3\": container with ID starting with 1cd34eeb1a7434b98e3641389e2f735ea7286d4530d2abe49975b8356e0dd3f3 not found: ID does not exist"
Sep 18 20:28:59.510959 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.510918    2573 scope.go:117] "RemoveContainer" containerID="1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"
Sep 18 20:28:59.511244 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:59.511224    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525\": container with ID starting with 1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525 not found: ID does not exist" containerID="1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"
Sep 18 20:28:59.511293 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.511252    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525"} err="failed to get container status \"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525\": rpc error: code = NotFound desc = could not find container \"1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525\": container with ID starting with 1e6e43dfc54ac102e43469da98be8bce15da5d23b55ab7e16d6ed928ac96d525 not found: ID does not exist"
Sep 18 20:28:59.511293 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.511270    2573 scope.go:117] "RemoveContainer" containerID="00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"
Sep 18 20:28:59.511514 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:28:59.511498    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017\": container with ID starting with 00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017 not found: ID does not exist" containerID="00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"
Sep 18 20:28:59.511564 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.511519    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017"} err="failed to get container status \"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017\": rpc error: code = NotFound desc = could not find container \"00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017\": container with ID starting with 00fb64c6969345963c54ed4dfc481e3c3dccf493d2d5787186a1ed739f627017 not found: ID does not exist"
Sep 18 20:28:59.625535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.625505    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") pod \"0d487173-29cf-4a30-9026-e217427195c3\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") "
Sep 18 20:28:59.625695 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.625561    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config\") pod \"0d487173-29cf-4a30-9026-e217427195c3\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") "
Sep 18 20:28:59.625695 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.625607    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location\") pod \"0d487173-29cf-4a30-9026-e217427195c3\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") "
Sep 18 20:28:59.625695 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.625688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert\") pod \"0d487173-29cf-4a30-9026-e217427195c3\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") "
Sep 18 20:28:59.625849 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.625722    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8fxv\" (UniqueName: \"kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv\") pod \"0d487173-29cf-4a30-9026-e217427195c3\" (UID: \"0d487173-29cf-4a30-9026-e217427195c3\") "
Sep 18 20:28:59.626063 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.626030    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d487173-29cf-4a30-9026-e217427195c3" (UID: "0d487173-29cf-4a30-9026-e217427195c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:28:59.626063 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.626048    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d487173-29cf-4a30-9026-e217427195c3" (UID: "0d487173-29cf-4a30-9026-e217427195c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:28:59.626195 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.626038    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d487173-29cf-4a30-9026-e217427195c3" (UID: "0d487173-29cf-4a30-9026-e217427195c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:28:59.627702 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.627681    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d487173-29cf-4a30-9026-e217427195c3" (UID: "0d487173-29cf-4a30-9026-e217427195c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:28:59.627807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.627791    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv" (OuterVolumeSpecName: "kube-api-access-g8fxv") pod "0d487173-29cf-4a30-9026-e217427195c3" (UID: "0d487173-29cf-4a30-9026-e217427195c3"). InnerVolumeSpecName "kube-api-access-g8fxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:28:59.726510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.726477    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:59.726510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.726505    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fxv\" (UniqueName: \"kubernetes.io/projected/0d487173-29cf-4a30-9026-e217427195c3-kube-api-access-g8fxv\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:59.726510 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.726515    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d487173-29cf-4a30-9026-e217427195c3-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:59.726732 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.726524    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d487173-29cf-4a30-9026-e217427195c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:59.726732 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.726532    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d487173-29cf-4a30-9026-e217427195c3-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:28:59.812009 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.811977    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:28:59.815767 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:28:59.815742    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d8vmz"]
Sep 18 20:29:01.317239 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:01.317185    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:29:01.782605 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:01.782576    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d487173-29cf-4a30-9026-e217427195c3" path="/var/lib/kubelet/pods/0d487173-29cf-4a30-9026-e217427195c3/volumes"
Sep 18 20:29:11.317218 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:11.317173    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:29:21.316912 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:21.316871    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:29:29.778409 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:29.778369    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:29:39.778736 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:39.778696    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:29:49.783626 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:49.783594    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:29:59.652266 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:59.652227    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:29:59.652723 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:59.652497    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" containerID="cri-o://9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37" gracePeriod=30
Sep 18 20:29:59.652723 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:59.652526    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kube-rbac-proxy" containerID="cri-o://b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb" gracePeriod=30
Sep 18 20:29:59.778300 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:29:59.778274    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Sep 18 20:30:00.739512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:00.739477    2573 generic.go:356] "Generic (PLEG): container finished" podID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerID="b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb" exitCode=2
Sep 18 20:30:00.739877 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:00.739525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerDied","Data":"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"}
Sep 18 20:30:01.311681 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:01.311635    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.71:8643/healthz\": dial tcp 10.134.0.71:8643: connect: connection refused"
Sep 18 20:30:03.995405 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:03.995379    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:30:04.130315 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert\") pod \"f958e977-bf39-45e4-a6b0-c16ab7018822\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") "
Sep 18 20:30:04.130578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130330    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config\") pod \"f958e977-bf39-45e4-a6b0-c16ab7018822\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") "
Sep 18 20:30:04.130578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130388    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qrlj\" (UniqueName: \"kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj\") pod \"f958e977-bf39-45e4-a6b0-c16ab7018822\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") "
Sep 18 20:30:04.130578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130426    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location\") pod \"f958e977-bf39-45e4-a6b0-c16ab7018822\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") "
Sep 18 20:30:04.130578 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130445    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls\") pod \"f958e977-bf39-45e4-a6b0-c16ab7018822\" (UID: \"f958e977-bf39-45e4-a6b0-c16ab7018822\") "
Sep 18 20:30:04.130807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130700    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f958e977-bf39-45e4-a6b0-c16ab7018822" (UID: "f958e977-bf39-45e4-a6b0-c16ab7018822"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:04.130807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130711    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f958e977-bf39-45e4-a6b0-c16ab7018822" (UID: "f958e977-bf39-45e4-a6b0-c16ab7018822"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:30:04.130807 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.130704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f958e977-bf39-45e4-a6b0-c16ab7018822" (UID: "f958e977-bf39-45e4-a6b0-c16ab7018822"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:30:04.132549 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.132525    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f958e977-bf39-45e4-a6b0-c16ab7018822" (UID: "f958e977-bf39-45e4-a6b0-c16ab7018822"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:30:04.132641 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.132623    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj" (OuterVolumeSpecName: "kube-api-access-4qrlj") pod "f958e977-bf39-45e4-a6b0-c16ab7018822" (UID: "f958e977-bf39-45e4-a6b0-c16ab7018822"). InnerVolumeSpecName "kube-api-access-4qrlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:30:04.231947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.231902    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrlj\" (UniqueName: \"kubernetes.io/projected/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-api-access-4qrlj\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:30:04.231947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.231942    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f958e977-bf39-45e4-a6b0-c16ab7018822-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:30:04.231947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.231952    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f958e977-bf39-45e4-a6b0-c16ab7018822-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:30:04.231947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.231961    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:30:04.232247 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.231970    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f958e977-bf39-45e4-a6b0-c16ab7018822-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:30:04.760344 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.760299    2573 generic.go:356] "Generic (PLEG): container finished" podID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerID="9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37" exitCode=0
Sep 18 20:30:04.760533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.760365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerDied","Data":"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"}
Sep 18 20:30:04.760533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.760376    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"
Sep 18 20:30:04.760533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.760432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w" event={"ID":"f958e977-bf39-45e4-a6b0-c16ab7018822","Type":"ContainerDied","Data":"73aca87999d0abab4ccb2273cec176bc2b3636f78d148d5446d2402538fd4de0"}
Sep 18 20:30:04.760533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.760468    2573 scope.go:117] "RemoveContainer" containerID="b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"
Sep 18 20:30:04.769463 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.769441    2573 scope.go:117] "RemoveContainer" containerID="9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"
Sep 18 20:30:04.777126 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.777104    2573 scope.go:117] "RemoveContainer" containerID="a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"
Sep 18 20:30:04.782526 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.782501    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:30:04.785159 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.785144    2573 scope.go:117] "RemoveContainer" containerID="b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"
Sep 18 20:30:04.785474 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:30:04.785452    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb\": container with ID starting with b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb not found: ID does not exist" containerID="b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"
Sep 18 20:30:04.785533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.785490    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb"} err="failed to get container status \"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb\": rpc error: code = NotFound desc = could not find container \"b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb\": container with ID starting with b3fabf886d3e57be06fe86ba7dbb99e440b5d09eb23a6164efaec660a846d1eb not found: ID does not exist"
Sep 18 20:30:04.785533 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.785507    2573 scope.go:117] "RemoveContainer" containerID="9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"
Sep 18 20:30:04.785791 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:30:04.785777    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37\": container with ID starting with 9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37 not found: ID does not exist" containerID="9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"
Sep 18 20:30:04.785829 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.785795    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37"} err="failed to get container status \"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37\": rpc error: code = NotFound desc = could not find container \"9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37\": container with ID starting with 9616b3e965a3b4a33fca85deaca8cd95fbcf10461bcbf09b17feb65940506d37 not found: ID does not exist"
Sep 18 20:30:04.785829 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.785808    2573 scope.go:117] "RemoveContainer" containerID="a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"
Sep 18 20:30:04.786039 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:30:04.786020    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382\": container with ID starting with a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382 not found: ID does not exist" containerID="a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"
Sep 18 20:30:04.786079 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.786048    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382"} err="failed to get container status \"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382\": rpc error: code = NotFound desc = could not find container \"a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382\": container with ID starting with a710b19fce54871669b83b1573a14579bd84c0dc44d0584ca82a828b2f04c382 not found: ID does not exist"
Sep 18 20:30:04.788702 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.788684    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hvx2w"]
Sep 18 20:30:04.844897 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.844868    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:30:04.845169 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845157    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845171    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845179    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kube-rbac-proxy"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845187    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kube-rbac-proxy"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845209    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845215    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845220    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kube-rbac-proxy"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845227    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kube-rbac-proxy"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845242    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="storage-initializer"
Sep 18 20:30:04.845246 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845248    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="storage-initializer"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845260    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="storage-initializer"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845265    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="storage-initializer"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845307    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kserve-container"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845317    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kserve-container"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845324    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" containerName="kube-rbac-proxy"
Sep 18 20:30:04.845522 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.845330    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d487173-29cf-4a30-9026-e217427195c3" containerName="kube-rbac-proxy"
Sep 18 20:30:04.850155 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.850134    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:04.852591 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852566    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:30:04.852725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852612    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:30:04.852725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852655    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:30:04.852858 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852747    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-274bef-predictor-serving-cert\""
Sep 18 20:30:04.852915 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852894    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pfnnj\""
Sep 18 20:30:04.852983 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852944    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-274bef-kube-rbac-proxy-sar-config\""
Sep 18 20:30:04.852983 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.852954    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:30:04.864690 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.864659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:30:04.937487 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.937451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:04.937678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.937511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:04.937678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.937550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:04.937678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.937625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:04.937678 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:04.937658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ds5r\" (UniqueName: \"kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.038805 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.038719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.038805 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.038760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ds5r\" (UniqueName: \"kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.038805 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.038795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.039330 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.038850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.039330 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.038874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.039330 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:30:05.038876    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-274bef-predictor-serving-cert: secret "isvc-primary-274bef-predictor-serving-cert" not found
Sep 18 20:30:05.039330 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:30:05.038966    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls podName:15650be8-e32b-4a7d-9441-4feeb5fcb02f nodeName:}" failed. No retries permitted until 2026-09-18 20:30:05.53894436 +0000 UTC m=+3114.056913978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls") pod "isvc-primary-274bef-predictor-57b4586b86-zd9wv" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f") : secret "isvc-primary-274bef-predictor-serving-cert" not found
Sep 18 20:30:05.039330 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.039313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.039506 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.039472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.039506 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.039498    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.061614 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.061585    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ds5r\" (UniqueName: \"kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.543441 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.543400    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.545977 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.545949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") pod \"isvc-primary-274bef-predictor-57b4586b86-zd9wv\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") " pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.766951 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.766918    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:05.783501 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:05.783465    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f958e977-bf39-45e4-a6b0-c16ab7018822" path="/var/lib/kubelet/pods/f958e977-bf39-45e4-a6b0-c16ab7018822/volumes"
Sep 18 20:30:06.036907 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:06.036845    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:30:06.039547 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:30:06.039518    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15650be8_e32b_4a7d_9441_4feeb5fcb02f.slice/crio-62ef243f99d52ff2e5335dd993ec1ad73b9b275249e0334ccb75b6e8d3b124c9 WatchSource:0}: Error finding container 62ef243f99d52ff2e5335dd993ec1ad73b9b275249e0334ccb75b6e8d3b124c9: Status 404 returned error can't find the container with id 62ef243f99d52ff2e5335dd993ec1ad73b9b275249e0334ccb75b6e8d3b124c9
Sep 18 20:30:06.769745 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:06.769702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerStarted","Data":"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"}
Sep 18 20:30:06.769947 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:06.769760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerStarted","Data":"62ef243f99d52ff2e5335dd993ec1ad73b9b275249e0334ccb75b6e8d3b124c9"}
Sep 18 20:30:10.786677 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:10.786649    2573 generic.go:356] "Generic (PLEG): container finished" podID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerID="815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a" exitCode=0
Sep 18 20:30:10.787123 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:10.786725    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerDied","Data":"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"}
Sep 18 20:30:11.793373 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:11.793337    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerStarted","Data":"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"}
Sep 18 20:30:11.793760 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:11.793396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerStarted","Data":"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"}
Sep 18 20:30:11.793760 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:11.793594    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:11.813278 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:11.813231    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podStartSLOduration=4.031259134 podStartE2EDuration="7.813216615s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.781957481s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:30:11.813023836 +0000 UTC m=+3120.330993489" watchObservedRunningTime="2026-09-18 20:30:11.813216615 +0000 UTC m=+3120.331186247"
Sep 18 20:30:12.797161 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:12.797116    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:12.798374 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:12.798344    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:13.800720 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:13.800684    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:18.805060 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:18.805035    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:30:18.805512 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:18.805488    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:28.806120 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:28.806034    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:38.805996 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:38.805957    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:48.805431 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:48.805388    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:30:58.806440 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:30:58.806399    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:31:08.806004 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:08.805964    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:31:18.805717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:18.805680    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 18 20:31:28.806377 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:28.806337    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:31:50.054110 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.054075    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:31:50.057618 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.057598    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.060446 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.060424    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-274bef-dockercfg-xrs2k\""
Sep 18 20:31:50.060633 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.060610    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-274bef-predictor-serving-cert\""
Sep 18 20:31:50.060900 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.060881    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-274bef\""
Sep 18 20:31:50.060945 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.060899    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-274bef-kube-rbac-proxy-sar-config\""
Sep 18 20:31:50.067691 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.067668    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:31:50.170828 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.170800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.170976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.170841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6kc\" (UniqueName: \"kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.170976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.170875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.170976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.170899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.170976 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.170914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.271457 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.271637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6kc\" (UniqueName: \"kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.271637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.271637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.271637 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.272010 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.271977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.272216 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.272179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.272289 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.272273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.274119 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.274092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.279729 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.279704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6kc\" (UniqueName: \"kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc\") pod \"isvc-secondary-274bef-predictor-6665d79445-nw7q9\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") " pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.368162 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.368129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:31:50.639423 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:50.639399    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:31:50.641614 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:31:50.641590    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0f002b_95a9_4e9a_abbf_661db6649a1f.slice/crio-9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6 WatchSource:0}: Error finding container 9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6: Status 404 returned error can't find the container with id 9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6
Sep 18 20:31:51.198535 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:51.198498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerStarted","Data":"474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"}
Sep 18 20:31:51.199048 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:51.198558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerStarted","Data":"9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6"}
Sep 18 20:31:53.207155 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:53.207124    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/0.log"
Sep 18 20:31:53.207560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:53.207163    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerID="474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5" exitCode=1
Sep 18 20:31:53.207560 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:53.207212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerDied","Data":"474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"}
Sep 18 20:31:54.212881 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:54.212852    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/0.log"
Sep 18 20:31:54.213261 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:54.212911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerStarted","Data":"8ee71c268cd9ecb528b5b652c85da8cbb90ea94c073a088cd8e6f4aa4d617977"}
Sep 18 20:31:56.221900 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.221877    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/1.log"
Sep 18 20:31:56.222311 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.222295    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/0.log"
Sep 18 20:31:56.222371 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.222329    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerID="8ee71c268cd9ecb528b5b652c85da8cbb90ea94c073a088cd8e6f4aa4d617977" exitCode=1
Sep 18 20:31:56.222416 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.222378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerDied","Data":"8ee71c268cd9ecb528b5b652c85da8cbb90ea94c073a088cd8e6f4aa4d617977"}
Sep 18 20:31:56.222460 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.222433    2573 scope.go:117] "RemoveContainer" containerID="474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"
Sep 18 20:31:56.222806 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.222773    2573 scope.go:117] "RemoveContainer" containerID="474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"
Sep 18 20:31:56.232827 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:31:56.232806    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-274bef-predictor-6665d79445-nw7q9_kserve-ci-e2e-test(3f0f002b-95a9-4e9a-abbf-661db6649a1f)\"" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f"
Sep 18 20:31:56.232997 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:31:56.232971    2573 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-274bef-predictor-6665d79445-nw7q9_kserve-ci-e2e-test_3f0f002b-95a9-4e9a-abbf-661db6649a1f_0 in pod sandbox 9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6 from index: no such id: '474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5'" containerID="474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"
Sep 18 20:31:56.233074 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:56.233007    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-274bef-predictor-6665d79445-nw7q9_kserve-ci-e2e-test_3f0f002b-95a9-4e9a-abbf-661db6649a1f_0 in pod sandbox 9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6 from index: no such id: '474b8bcd778ef4933b118641f48c7e102ca50221ac498bd767ce47bb4228c5c5'"
Sep 18 20:31:57.228586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:31:57.228560    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/1.log"
Sep 18 20:32:04.043534 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.043500    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:32:04.069268 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.069234    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:32:04.069557 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.069528    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container" containerID="cri-o://f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc" gracePeriod=30
Sep 18 20:32:04.069634 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.069556    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kube-rbac-proxy" containerID="cri-o://95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d" gracePeriod=30
Sep 18 20:32:04.189324 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.189305    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/1.log"
Sep 18 20:32:04.191411 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.191397    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:32:04.261608 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.261577    2573 generic.go:356] "Generic (PLEG): container finished" podID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerID="95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d" exitCode=2
Sep 18 20:32:04.261782 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.261647    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerDied","Data":"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"}
Sep 18 20:32:04.262918 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.262901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-274bef-predictor-6665d79445-nw7q9_3f0f002b-95a9-4e9a-abbf-661db6649a1f/storage-initializer/1.log"
Sep 18 20:32:04.263025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.262954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9" event={"ID":"3f0f002b-95a9-4e9a-abbf-661db6649a1f","Type":"ContainerDied","Data":"9fd71942ea1b1a4a4aec571d1218212914a4dda132d535f3825c7f88d2545ec6"}
Sep 18 20:32:04.263025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.262988    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"
Sep 18 20:32:04.263025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.263010    2573 scope.go:117] "RemoveContainer" containerID="8ee71c268cd9ecb528b5b652c85da8cbb90ea94c073a088cd8e6f4aa4d617977"
Sep 18 20:32:04.285080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285055    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv6kc\" (UniqueName: \"kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc\") pod \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") "
Sep 18 20:32:04.285182 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285094    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config\") pod \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") "
Sep 18 20:32:04.285182 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285132    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert\") pod \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") "
Sep 18 20:32:04.285335 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285242    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls\") pod \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") "
Sep 18 20:32:04.285335 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location\") pod \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\" (UID: \"3f0f002b-95a9-4e9a-abbf-661db6649a1f\") "
Sep 18 20:32:04.285524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285482    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f0f002b-95a9-4e9a-abbf-661db6649a1f" (UID: "3f0f002b-95a9-4e9a-abbf-661db6649a1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:04.285619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f0f002b-95a9-4e9a-abbf-661db6649a1f" (UID: "3f0f002b-95a9-4e9a-abbf-661db6649a1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:04.285619 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.285504    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f0f002b-95a9-4e9a-abbf-661db6649a1f" (UID: "3f0f002b-95a9-4e9a-abbf-661db6649a1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:04.287141 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.287112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f0f002b-95a9-4e9a-abbf-661db6649a1f" (UID: "3f0f002b-95a9-4e9a-abbf-661db6649a1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:04.287141 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.287130    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc" (OuterVolumeSpecName: "kube-api-access-tv6kc") pod "3f0f002b-95a9-4e9a-abbf-661db6649a1f" (UID: "3f0f002b-95a9-4e9a-abbf-661db6649a1f"). InnerVolumeSpecName "kube-api-access-tv6kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:04.386762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.386728    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.386762 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.386758    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f002b-95a9-4e9a-abbf-661db6649a1f-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.386957 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.386771    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f0f002b-95a9-4e9a-abbf-661db6649a1f-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.386957 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.386783    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.386957 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.386795    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv6kc\" (UniqueName: \"kubernetes.io/projected/3f0f002b-95a9-4e9a-abbf-661db6649a1f-kube-api-access-tv6kc\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:04.603029 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.602999    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:32:04.605494 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:04.605468    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-274bef-predictor-6665d79445-nw7q9"]
Sep 18 20:32:05.782339 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:05.782297    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" path="/var/lib/kubelet/pods/3f0f002b-95a9-4e9a-abbf-661db6649a1f/volumes"
Sep 18 20:32:08.415537 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.415513    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:32:08.523440 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523355    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config\") pod \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") "
Sep 18 20:32:08.523440 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ds5r\" (UniqueName: \"kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r\") pod \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") "
Sep 18 20:32:08.523440 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523436    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") pod \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") "
Sep 18 20:32:08.523729 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523542    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location\") pod \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") "
Sep 18 20:32:08.523729 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523608    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert\") pod \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\" (UID: \"15650be8-e32b-4a7d-9441-4feeb5fcb02f\") "
Sep 18 20:32:08.523879 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523841    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15650be8-e32b-4a7d-9441-4feeb5fcb02f" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:08.524000 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523851    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15650be8-e32b-4a7d-9441-4feeb5fcb02f" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:08.524000 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.523923    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15650be8-e32b-4a7d-9441-4feeb5fcb02f" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:32:08.525586 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.525557    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r" (OuterVolumeSpecName: "kube-api-access-6ds5r") pod "15650be8-e32b-4a7d-9441-4feeb5fcb02f" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f"). InnerVolumeSpecName "kube-api-access-6ds5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:08.525672 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.525596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15650be8-e32b-4a7d-9441-4feeb5fcb02f" (UID: "15650be8-e32b-4a7d-9441-4feeb5fcb02f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:32:08.624211 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.624183    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kserve-provision-location\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:08.624323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.624225    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-cabundle-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:08.624323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.624240    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:08.624323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.624253    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ds5r\" (UniqueName: \"kubernetes.io/projected/15650be8-e32b-4a7d-9441-4feeb5fcb02f-kube-api-access-6ds5r\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:08.624323 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:08.624266    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15650be8-e32b-4a7d-9441-4feeb5fcb02f-proxy-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:09.284604 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.284568    2573 generic.go:356] "Generic (PLEG): container finished" podID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerID="f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc" exitCode=0
Sep 18 20:32:09.284794 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.284630    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"
Sep 18 20:32:09.284794 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.284634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerDied","Data":"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"}
Sep 18 20:32:09.284794 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.284709    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv" event={"ID":"15650be8-e32b-4a7d-9441-4feeb5fcb02f","Type":"ContainerDied","Data":"62ef243f99d52ff2e5335dd993ec1ad73b9b275249e0334ccb75b6e8d3b124c9"}
Sep 18 20:32:09.284794 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.284750    2573 scope.go:117] "RemoveContainer" containerID="95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"
Sep 18 20:32:09.294034 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.293839    2573 scope.go:117] "RemoveContainer" containerID="f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"
Sep 18 20:32:09.301329 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.301308    2573 scope.go:117] "RemoveContainer" containerID="815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"
Sep 18 20:32:09.306861 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.306830    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:32:09.308283 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.308258    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-274bef-predictor-57b4586b86-zd9wv"]
Sep 18 20:32:09.310019 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.309990    2573 scope.go:117] "RemoveContainer" containerID="95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"
Sep 18 20:32:09.310339 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:09.310320    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d\": container with ID starting with 95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d not found: ID does not exist" containerID="95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"
Sep 18 20:32:09.310413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.310349    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d"} err="failed to get container status \"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d\": rpc error: code = NotFound desc = could not find container \"95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d\": container with ID starting with 95cd34d42d71dacb4637addedaee33cbbea3c053ac5b81790c8fc77f6eafbc4d not found: ID does not exist"
Sep 18 20:32:09.310413 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.310373    2573 scope.go:117] "RemoveContainer" containerID="f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"
Sep 18 20:32:09.310669 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:09.310641    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc\": container with ID starting with f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc not found: ID does not exist" containerID="f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"
Sep 18 20:32:09.310725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.310666    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc"} err="failed to get container status \"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc\": rpc error: code = NotFound desc = could not find container \"f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc\": container with ID starting with f2f9dd7ed9b35e2f3ff8468e1d8250a32c0a04350c1eed34cd76cef3579e1cfc not found: ID does not exist"
Sep 18 20:32:09.310725 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.310685    2573 scope.go:117] "RemoveContainer" containerID="815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"
Sep 18 20:32:09.310945 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:09.310928    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a\": container with ID starting with 815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a not found: ID does not exist" containerID="815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"
Sep 18 20:32:09.310983 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.310952    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a"} err="failed to get container status \"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a\": rpc error: code = NotFound desc = could not find container \"815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a\": container with ID starting with 815e3be7828d92a36b1899abed7e12dfe4a6f0c2b9d455c1d73d4e36ef6adf3a not found: ID does not exist"
Sep 18 20:32:09.786491 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:09.783177    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" path="/var/lib/kubelet/pods/15650be8-e32b-4a7d-9441-4feeb5fcb02f/volumes"
Sep 18 20:32:10.429089 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429055    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-df48p/must-gather-c6cs6"]
Sep 18 20:32:10.429381 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429367    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="storage-initializer"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429382    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="storage-initializer"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429397    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kube-rbac-proxy"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429402    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kube-rbac-proxy"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429415    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429421    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429428    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container"
Sep 18 20:32:10.429438 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429435    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429480    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kube-rbac-proxy"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429490    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="15650be8-e32b-4a7d-9441-4feeb5fcb02f" containerName="kserve-container"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429496    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429551    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429556    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.429656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.429608    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0f002b-95a9-4e9a-abbf-661db6649a1f" containerName="storage-initializer"
Sep 18 20:32:10.434053 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.434033    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.436506 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.436483    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-df48p\"/\"openshift-service-ca.crt\""
Sep 18 20:32:10.436506 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.436485    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-df48p\"/\"kube-root-ca.crt\""
Sep 18 20:32:10.436683 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.436513    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-df48p\"/\"default-dockercfg-rf9b9\""
Sep 18 20:32:10.441822 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.441801    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-df48p/must-gather-c6cs6"]
Sep 18 20:32:10.538953 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.538895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.539119 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.538968    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjlx8\" (UniqueName: \"kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.639452 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.639422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjlx8\" (UniqueName: \"kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.639599 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.639476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.639754 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.639740    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.647269 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.647249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjlx8\" (UniqueName: \"kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8\") pod \"must-gather-c6cs6\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") " pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:10.751466 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:10.751407    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:11.018635 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:11.018557    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-df48p/must-gather-c6cs6"]
Sep 18 20:32:11.021462 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:32:11.021434    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7db3d46_058d_4328_ac5d_6a4937c56279.slice/crio-c995902b4149cef63cd7bdf8ebc6995db25c73682dc458e5964a660488d543e0 WatchSource:0}: Error finding container c995902b4149cef63cd7bdf8ebc6995db25c73682dc458e5964a660488d543e0: Status 404 returned error can't find the container with id c995902b4149cef63cd7bdf8ebc6995db25c73682dc458e5964a660488d543e0
Sep 18 20:32:11.023236 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:11.023220    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:32:11.294424 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:11.294338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-df48p/must-gather-c6cs6" event={"ID":"e7db3d46-058d-4328-ac5d-6a4937c56279","Type":"ContainerStarted","Data":"c995902b4149cef63cd7bdf8ebc6995db25c73682dc458e5964a660488d543e0"}
Sep 18 20:32:17.328573 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:17.328520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-df48p/must-gather-c6cs6" event={"ID":"e7db3d46-058d-4328-ac5d-6a4937c56279","Type":"ContainerStarted","Data":"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"}
Sep 18 20:32:18.334704 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:18.334664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-df48p/must-gather-c6cs6" event={"ID":"e7db3d46-058d-4328-ac5d-6a4937c56279","Type":"ContainerStarted","Data":"855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344"}
Sep 18 20:32:18.350252 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:18.350171    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-df48p/must-gather-c6cs6" podStartSLOduration=2.241278005 podStartE2EDuration="8.350152338s" totalImagesPullingTime="6.108874333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:18.34935594 +0000 UTC m=+3246.867325581" watchObservedRunningTime="2026-09-18 20:32:18.350152338 +0000 UTC m=+3246.868121979"
Sep 18 20:32:43.447500 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:43.447415    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerID="9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce" exitCode=0
Sep 18 20:32:43.447500 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:43.447488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-df48p/must-gather-c6cs6" event={"ID":"e7db3d46-058d-4328-ac5d-6a4937c56279","Type":"ContainerDied","Data":"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"}
Sep 18 20:32:43.447952 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:43.447802    2573 scope.go:117] "RemoveContainer" containerID="9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"
Sep 18 20:32:44.206791 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.206757    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-df48p_must-gather-c6cs6_e7db3d46-058d-4328-ac5d-6a4937c56279/gather/0.log"
Sep 18 20:32:44.867910 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.867880    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cdvqj/must-gather-fwnxs"]
Sep 18 20:32:44.871142 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.871126    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:44.873327 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.873304    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdvqj\"/\"openshift-service-ca.crt\""
Sep 18 20:32:44.873427 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.873338    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cdvqj\"/\"default-dockercfg-8c4rd\""
Sep 18 20:32:44.874155 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.874136    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cdvqj\"/\"kube-root-ca.crt\""
Sep 18 20:32:44.879270 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.879252    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/must-gather-fwnxs"]
Sep 18 20:32:44.933045 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.933003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/557d45ed-dde8-426f-959d-fe3d583b51ce-must-gather-output\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:44.933304 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:44.933111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5xj\" (UniqueName: \"kubernetes.io/projected/557d45ed-dde8-426f-959d-fe3d583b51ce-kube-api-access-7q5xj\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.034354 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.034317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5xj\" (UniqueName: \"kubernetes.io/projected/557d45ed-dde8-426f-959d-fe3d583b51ce-kube-api-access-7q5xj\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.034525 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.034385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/557d45ed-dde8-426f-959d-fe3d583b51ce-must-gather-output\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.034675 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.034661    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/557d45ed-dde8-426f-959d-fe3d583b51ce-must-gather-output\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.042484 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.042458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5xj\" (UniqueName: \"kubernetes.io/projected/557d45ed-dde8-426f-959d-fe3d583b51ce-kube-api-access-7q5xj\") pod \"must-gather-fwnxs\" (UID: \"557d45ed-dde8-426f-959d-fe3d583b51ce\") " pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.181022 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.180937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/must-gather-fwnxs"
Sep 18 20:32:45.446868 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.446843    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/must-gather-fwnxs"]
Sep 18 20:32:45.448806 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:32:45.448776    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod557d45ed_dde8_426f_959d_fe3d583b51ce.slice/crio-e58ecec89d01b09d7dea3401e0336998789ee8d33fc0b4688e657ad15a268057 WatchSource:0}: Error finding container e58ecec89d01b09d7dea3401e0336998789ee8d33fc0b4688e657ad15a268057: Status 404 returned error can't find the container with id e58ecec89d01b09d7dea3401e0336998789ee8d33fc0b4688e657ad15a268057
Sep 18 20:32:45.458265 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:45.458239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/must-gather-fwnxs" event={"ID":"557d45ed-dde8-426f-959d-fe3d583b51ce","Type":"ContainerStarted","Data":"e58ecec89d01b09d7dea3401e0336998789ee8d33fc0b4688e657ad15a268057"}
Sep 18 20:32:46.465652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:46.465566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/must-gather-fwnxs" event={"ID":"557d45ed-dde8-426f-959d-fe3d583b51ce","Type":"ContainerStarted","Data":"723813ffcdf2244e5539f1c558e482d3c71da4073bc43c227bf5070c10327b07"}
Sep 18 20:32:46.465652 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:46.465617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/must-gather-fwnxs" event={"ID":"557d45ed-dde8-426f-959d-fe3d583b51ce","Type":"ContainerStarted","Data":"07568a7721b5ab04a55f2e2bef820559f83059ae070210da036e0d4b3d0f6c51"}
Sep 18 20:32:46.481526 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:46.481467    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cdvqj/must-gather-fwnxs" podStartSLOduration=1.72841297 podStartE2EDuration="2.481447686s" totalImagesPullingTime="753.034716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:46.48060338 +0000 UTC m=+3274.998573022" watchObservedRunningTime="2026-09-18 20:32:46.481447686 +0000 UTC m=+3274.999417327"
Sep 18 20:32:47.823403 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:47.823359    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f75sh_a8290d9b-8eba-4ca6-9452-1a9fcf6e8a4d/global-pull-secret-syncer/0.log"
Sep 18 20:32:47.996895 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:47.996866    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fwbh5_6e010f81-8cd6-4365-83df-7651a560fba1/konnectivity-agent/0.log"
Sep 18 20:32:48.018686 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:48.018648    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-207.ec2.internal_17a176fdef57f822539a4bd26e5f2da2/haproxy/0.log"
Sep 18 20:32:49.724049 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:49.724013    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-df48p/must-gather-c6cs6"]
Sep 18 20:32:49.724831 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:49.724798    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-df48p/must-gather-c6cs6" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="copy" containerID="cri-o://855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344" gracePeriod=2
Sep 18 20:32:49.727036 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:49.726886    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-c6cs6\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-df48p\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:49.730257 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:49.730238    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-df48p/must-gather-c6cs6"]
Sep 18 20:32:50.101094 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.101056    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-df48p_must-gather-c6cs6_e7db3d46-058d-4328-ac5d-6a4937c56279/copy/0.log"
Sep 18 20:32:50.106980 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.106338    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:50.183689 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.183651    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output\") pod \"e7db3d46-058d-4328-ac5d-6a4937c56279\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") "
Sep 18 20:32:50.183890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.183714    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjlx8\" (UniqueName: \"kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8\") pod \"e7db3d46-058d-4328-ac5d-6a4937c56279\" (UID: \"e7db3d46-058d-4328-ac5d-6a4937c56279\") "
Sep 18 20:32:50.185791 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.185720    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e7db3d46-058d-4328-ac5d-6a4937c56279" (UID: "e7db3d46-058d-4328-ac5d-6a4937c56279"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:32:50.190546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.190490    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8" (OuterVolumeSpecName: "kube-api-access-pjlx8") pod "e7db3d46-058d-4328-ac5d-6a4937c56279" (UID: "e7db3d46-058d-4328-ac5d-6a4937c56279"). InnerVolumeSpecName "kube-api-access-pjlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:32:50.285027 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.284958    2573 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7db3d46-058d-4328-ac5d-6a4937c56279-must-gather-output\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:50.285027 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.284997    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjlx8\" (UniqueName: \"kubernetes.io/projected/e7db3d46-058d-4328-ac5d-6a4937c56279-kube-api-access-pjlx8\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\""
Sep 18 20:32:50.491079 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.490993    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-df48p_must-gather-c6cs6_e7db3d46-058d-4328-ac5d-6a4937c56279/copy/0.log"
Sep 18 20:32:50.495459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.491623    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerID="855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344" exitCode=143
Sep 18 20:32:50.495459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.491749    2573 scope.go:117] "RemoveContainer" containerID="855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344"
Sep 18 20:32:50.495459 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.491911    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-df48p/must-gather-c6cs6"
Sep 18 20:32:50.522594 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.521296    2573 scope.go:117] "RemoveContainer" containerID="9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"
Sep 18 20:32:50.545100 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.544519    2573 scope.go:117] "RemoveContainer" containerID="855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344"
Sep 18 20:32:50.545100 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:50.544941    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344\": container with ID starting with 855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344 not found: ID does not exist" containerID="855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344"
Sep 18 20:32:50.545100 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.544982    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344"} err="failed to get container status \"855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344\": rpc error: code = NotFound desc = could not find container \"855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344\": container with ID starting with 855c6692de9c4bede99a89c1173c2aaef496e71bd77fac49e3764c542dc42344 not found: ID does not exist"
Sep 18 20:32:50.545100 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.545008    2573 scope.go:117] "RemoveContainer" containerID="9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"
Sep 18 20:32:50.545441 ip-10-0-136-207 kubenswrapper[2573]: E0918 20:32:50.545347    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce\": container with ID starting with 9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce not found: ID does not exist" containerID="9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"
Sep 18 20:32:50.545441 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:50.545381    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce"} err="failed to get container status \"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce\": rpc error: code = NotFound desc = could not find container \"9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce\": container with ID starting with 9fea5498c990662fd41e0efb36b1537d2576aeab6485a1b99890f2e38b88afce not found: ID does not exist"
Sep 18 20:32:51.784917 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:51.784878    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" path="/var/lib/kubelet/pods/e7db3d46-058d-4328-ac5d-6a4937c56279/volumes"
Sep 18 20:32:52.025224 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.023935    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/alertmanager/0.log"
Sep 18 20:32:52.066183 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.066105    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/config-reloader/0.log"
Sep 18 20:32:52.093466 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.093425    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/kube-rbac-proxy-web/0.log"
Sep 18 20:32:52.118089 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.118055    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/kube-rbac-proxy/0.log"
Sep 18 20:32:52.141550 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.141516    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/kube-rbac-proxy-metric/0.log"
Sep 18 20:32:52.166449 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.166310    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/prom-label-proxy/0.log"
Sep 18 20:32:52.186488 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.186462    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d743314e-1d70-4be3-9445-b2c9d9280bb2/init-config-reloader/0.log"
Sep 18 20:32:52.281970 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.281932    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-swcc4_95c56be6-2ba8-494c-b2a6-610c3e25fe4e/kube-state-metrics/0.log"
Sep 18 20:32:52.302877 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.302845    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-swcc4_95c56be6-2ba8-494c-b2a6-610c3e25fe4e/kube-rbac-proxy-main/0.log"
Sep 18 20:32:52.327910 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.327813    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-swcc4_95c56be6-2ba8-494c-b2a6-610c3e25fe4e/kube-rbac-proxy-self/0.log"
Sep 18 20:32:52.568147 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.568114    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlhrv_7962429b-25d2-4e9f-ad55-e8a01ab50822/node-exporter/0.log"
Sep 18 20:32:52.586520 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.586433    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlhrv_7962429b-25d2-4e9f-ad55-e8a01ab50822/kube-rbac-proxy/0.log"
Sep 18 20:32:52.614080 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.614058    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlhrv_7962429b-25d2-4e9f-ad55-e8a01ab50822/init-textfile/0.log"
Sep 18 20:32:52.894415 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.894384    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/prometheus/0.log"
Sep 18 20:32:52.910364 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.910328    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/config-reloader/0.log"
Sep 18 20:32:52.932248 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.932219    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/thanos-sidecar/0.log"
Sep 18 20:32:52.952816 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.952792    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/kube-rbac-proxy-web/0.log"
Sep 18 20:32:52.975239 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.975216    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/kube-rbac-proxy/0.log"
Sep 18 20:32:52.995876 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:52.995827    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/kube-rbac-proxy-thanos/0.log"
Sep 18 20:32:53.015162 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.015129    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e91a6d-98ca-453f-800d-a8f36f39518a/init-config-reloader/0.log"
Sep 18 20:32:53.328312 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.328282    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/thanos-query/0.log"
Sep 18 20:32:53.353761 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.353730    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/kube-rbac-proxy-web/0.log"
Sep 18 20:32:53.374249 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.374221    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/kube-rbac-proxy/0.log"
Sep 18 20:32:53.396717 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.396683    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/prom-label-proxy/0.log"
Sep 18 20:32:53.419385 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.419283    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/kube-rbac-proxy-rules/0.log"
Sep 18 20:32:53.468357 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:53.468326    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55c6b56b78-5q7rx_e5baf44d-2bd1-4deb-9c4e-91a7d71a6ea0/kube-rbac-proxy-metrics/0.log"
Sep 18 20:32:56.239940 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.239907    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"]
Sep 18 20:32:56.240521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240422    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="gather"
Sep 18 20:32:56.240521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240442    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="gather"
Sep 18 20:32:56.240521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240489    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="copy"
Sep 18 20:32:56.240521 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="copy"
Sep 18 20:32:56.240737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240560    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="copy"
Sep 18 20:32:56.240737 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.240579    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7db3d46-058d-4328-ac5d-6a4937c56279" containerName="gather"
Sep 18 20:32:56.246175 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.246144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.251656 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.251629    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"]
Sep 18 20:32:56.346835 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.346799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-proc\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.347023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.346848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-sys\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.347023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.346900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-podres\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.347023 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.346997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9ss\" (UniqueName: \"kubernetes.io/projected/474313c7-22a1-420a-8589-f8fcfd874b1f-kube-api-access-7m9ss\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.347232 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.347053    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-lib-modules\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448358 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-lib-modules\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-proc\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-sys\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-podres\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9ss\" (UniqueName: \"kubernetes.io/projected/474313c7-22a1-420a-8589-f8fcfd874b1f-kube-api-access-7m9ss\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448546 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-lib-modules\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448823 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448598    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-proc\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448823 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-sys\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.448823 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.448739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/474313c7-22a1-420a-8589-f8fcfd874b1f-podres\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.456168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.456129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9ss\" (UniqueName: \"kubernetes.io/projected/474313c7-22a1-420a-8589-f8fcfd874b1f-kube-api-access-7m9ss\") pod \"perf-node-gather-daemonset-nxrhp\" (UID: \"474313c7-22a1-420a-8589-f8fcfd874b1f\") " pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.559321 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.559237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:56.843869 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:56.842465    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"]
Sep 18 20:32:56.845594 ip-10-0-136-207 kubenswrapper[2573]: W0918 20:32:56.845560    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod474313c7_22a1_420a_8589_f8fcfd874b1f.slice/crio-9e77d6e2be23760b8ec080d61ea1fa59418efe31f80d19b40426d3dde307d192 WatchSource:0}: Error finding container 9e77d6e2be23760b8ec080d61ea1fa59418efe31f80d19b40426d3dde307d192: Status 404 returned error can't find the container with id 9e77d6e2be23760b8ec080d61ea1fa59418efe31f80d19b40426d3dde307d192
Sep 18 20:32:57.385135 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.385109    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzbnp_7a2b66dd-9172-492c-94cd-0f61949f8759/dns/0.log"
Sep 18 20:32:57.405758 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.405721    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mzbnp_7a2b66dd-9172-492c-94cd-0f61949f8759/kube-rbac-proxy/0.log"
Sep 18 20:32:57.428076 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.428041    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-29vx8_fd7438f0-23d8-42bd-82fc-fd43c3f98c1b/dns-node-resolver/0.log"
Sep 18 20:32:57.535643 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.535608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp" event={"ID":"474313c7-22a1-420a-8589-f8fcfd874b1f","Type":"ContainerStarted","Data":"257a95fb53d2b837f612f559afafdab4959228bc5bc675b8a60691e77e414928"}
Sep 18 20:32:57.535826 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.535670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp" event={"ID":"474313c7-22a1-420a-8589-f8fcfd874b1f","Type":"ContainerStarted","Data":"9e77d6e2be23760b8ec080d61ea1fa59418efe31f80d19b40426d3dde307d192"}
Sep 18 20:32:57.535890 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.535844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:32:57.962136 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:57.962105    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cxk6_0385842a-4021-45f6-9d3d-606a34d90b5e/node-ca/0.log"
Sep 18 20:32:59.087962 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:59.087934    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wgnm9_233b6f4b-32f4-4bf3-ad0b-878dfd95d556/serve-healthcheck-canary/0.log"
Sep 18 20:32:59.490184 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:59.490159    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64wpr_b1904fe7-40ac-46fe-a615-39ce797ef1c1/kube-rbac-proxy/0.log"
Sep 18 20:32:59.507751 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:59.507727    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64wpr_b1904fe7-40ac-46fe-a615-39ce797ef1c1/exporter/0.log"
Sep 18 20:32:59.527303 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:32:59.527281    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64wpr_b1904fe7-40ac-46fe-a615-39ce797ef1c1/extractor/0.log"
Sep 18 20:33:02.042168 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.042137    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-w9cj8_95ba16ae-9ee4-4667-a900-83fa6c0ce16e/server/0.log"
Sep 18 20:33:02.344082 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.344056    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-mhp2g_44e99a3c-f8fc-42dc-9bd8-2e8d5b3e6d03/manager/0.log"
Sep 18 20:33:02.382031 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.382005    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2krtg_ce6a702e-2068-45e0-81e9-ecb8e4f6411c/s3-init/0.log"
Sep 18 20:33:02.428157 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.428118    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2krtg_ce6a702e-2068-45e0-81e9-ecb8e4f6411c/download-hf-model/0.log"
Sep 18 20:33:02.452119 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.452089    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vbnhs_73acbcc9-eb22-4f8f-a638-6bab293eba24/s3-tls-init-custom/0.log"
Sep 18 20:33:02.466179 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.466156    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vbnhs_73acbcc9-eb22-4f8f-a638-6bab293eba24/download-hf-model/0.log"
Sep 18 20:33:02.492251 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.492225    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6s2f4_47486c0a-e2e9-4b68-b23c-04bffe84b6fa/s3-tls-init-serving/0.log"
Sep 18 20:33:02.508524 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.508499    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6s2f4_47486c0a-e2e9-4b68-b23c-04bffe84b6fa/download-hf-model/0.log"
Sep 18 20:33:02.532735 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:02.532711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2ghw8_cbeea091-d888-4b30-a394-30edd3e366c4/seaweedfs/0.log"
Sep 18 20:33:03.550025 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:03.549997    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp"
Sep 18 20:33:03.567686 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:03.567630    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cdvqj/perf-node-gather-daemonset-nxrhp" podStartSLOduration=7.567614299 podStartE2EDuration="7.567614299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:32:57.550722971 +0000 UTC m=+3286.068692614" watchObservedRunningTime="2026-09-18 20:33:03.567614299 +0000 UTC m=+3292.085583975"
Sep 18 20:33:08.027419 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.027391    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:33:08.046222 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.046132    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/egress-router-binary-copy/0.log"
Sep 18 20:33:08.064735 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.064711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/cni-plugins/0.log"
Sep 18 20:33:08.083390 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.083363    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/bond-cni-plugin/0.log"
Sep 18 20:33:08.103454 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.103425    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/routeoverride-cni/0.log"
Sep 18 20:33:08.123572 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.123549    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/whereabouts-cni-bincopy/0.log"
Sep 18 20:33:08.144706 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.144684    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx2s7_4e1f3ce1-13e9-4c68-afca-dc5e17918cac/whereabouts-cni/0.log"
Sep 18 20:33:08.365603 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.365574    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xcshw_44a30c5a-0fc4-4da3-bc2f-8a67d96fc74c/kube-multus/0.log"
Sep 18 20:33:08.434827 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.434799    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8grvn_87591871-d05d-4e65-ba24-915d085bf6b2/network-metrics-daemon/0.log"
Sep 18 20:33:08.501100 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:08.501073    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8grvn_87591871-d05d-4e65-ba24-915d085bf6b2/kube-rbac-proxy/0.log"
Sep 18 20:33:09.343267 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.343241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/ovn-controller/0.log"
Sep 18 20:33:09.357981 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.357954    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/ovn-acl-logging/0.log"
Sep 18 20:33:09.377066 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.377036    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/kube-rbac-proxy-node/0.log"
Sep 18 20:33:09.396655 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.396627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:33:09.414792 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.414767    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/northd/0.log"
Sep 18 20:33:09.434450 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.434427    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/nbdb/0.log"
Sep 18 20:33:09.455189 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.455164    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/sbdb/0.log"
Sep 18 20:33:09.514771 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:09.514744    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ds2kn_34becf8e-bd90-4e82-80c0-ff527bee92bd/ovnkube-controller/0.log"
Sep 18 20:33:10.950787 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:10.950757    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kh7cd_52504afc-712e-48f5-a8d4-586460e326c5/network-check-target-container/0.log"
Sep 18 20:33:11.866007 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:11.865973    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wmlkl_3a37a033-426d-4229-bebf-67a87e8d980a/iptables-alerter/0.log"
Sep 18 20:33:12.536778 ip-10-0-136-207 kubenswrapper[2573]: I0918 20:33:12.536738    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rwjqg_c46ec78c-24d5-4a08-9c45-f5b3cebc62dc/tuned/0.log"